System

Abstract

To provide a system for realizing efficient and effective progress of a conference by automating the whole process of an online conference.SOLUTION: Means for acquiring a scheduled date and time of a meeting, a list of participants, and an agenda, means for transmitting a reminder to a participant, means for generating a meeting link at a start time of the meeting and notifying the participant of the meeting link, means for activating AI moderation to receive a statement request and arrange a statement order, means for transcribing meeting content in real time, means for extracting a summary of the meeting content and sharing the summary with the participants, means for inputting a question into a chat, analyzing the question, and notifying the participants of the question, means for checking action items before the end of the meeting, A system comprising: means for notifying a participant; and means for sending a follow-up notification after a meeting ends.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Properly managing online meetings and ensuring smooth progress and active communication are difficult challenges. In particular, proper management of speakers, recording and sharing of meeting content, smooth handling of participant questions and answers, and follow-up after the meeting are often lacking. Furthermore, managing these processes manually is time-consuming and inefficient. Given these circumstances, we hope to solve these challenges by developing a system that utilizes AI. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for acquiring the scheduled date and time of a meeting, a participant list, and an agenda; means for sending reminders to participants; means for generating a meeting link at the start time of the meeting and notifying participants; means for activating an AI moderator, receiving speaking requests, and organizing the speaking order; means for transcribing the meeting content in real time; means for extracting a summary of the meeting content and sharing it with participants; means for entering questions into a chat, analyzing the questions, and notifying participants; means for checking action items before the end of the meeting and notifying participants; and means for sending follow-up notifications after the end of the meeting. This system automates the entire process of online meetings, enabling efficient and effective meeting conduct.

[0006] The "scheduled date and time of a conference" refers to the start time and end time of a conference that will be held on a specific date and time.

[0007] "Participant List" refers to a list of all members invited to an online meeting.

[0008] An "agenda" is a plan that lists the topics to be discussed in a meeting, the order of each item, and the time allocation.

[0009] "Reminder" refers to a notification method such as email or alert that notifies people in advance of a meeting.

[0010] "Conference Link" means the URL or other electronic access information used to access an online conference.

[0011] "AI moderation" refers to a system that uses artificial intelligence to manage the progress of a meeting and organize the order and content of speeches.

[0012] A "request to speak" refers to a participant's expression of intent to request permission to speak during a conference.

[0013] "Ordering speech" refers to the process of accepting multiple requests to speak and granting permission to speak in the appropriate order.

[0014] "Means of transcribing meeting content in real time" refers to the process of converting what is said during a meeting into text using speech recognition technology.

[0015] "Extracting a summary" refers to the process of extracting important points and key discussion points from a meeting and summarizing them in a concise format.

[0016] "Follow-up notifications" refer to a means of informing participants after a meeting about the progress of action items and uncompleted tasks. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting.

[0039] System programs and their processing

[0040] 1. Planning a meeting

[0041] First, the server retrieves the scheduled meeting date and time, the list of attendees, and the agenda. This information can be retrieved from a calendar API or a database. Based on this information, the server sends reminders to attendees. These reminders can be sent via email or a notification system.

[0042] 2. Starting the meeting

[0043] Next, when the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. When the meeting starts, the server launches an AI moderator to manage the progress of the meeting.

[0044] 3. Comment Management

[0045] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[0046] 4. Record and share content

[0047] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[0048] 5. Question and Answer Management

[0049] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[0050] 6. Review and follow up on action items

[0051] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[0052] Specific examples

[0053] Example 1: Planning a meeting

[0054] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday. With this information, the server sends reminders to participants via email, ensuring they don't forget the meeting date.

[0055] Example 2: Speech Management

[0056] During a meeting, a user sends a request to speak from their device. The device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will speak next."

[0057] Example 3: Managing questions and answers

[0058] During a meeting, a user can enter a question via chat. The server receives the question, and the AI ​​moderator responds by saying, "A new question has arrived. Mr. Sato, please answer." This function ensures a smooth flow of questions and answers.

[0059] In this way, the system of the present invention can automate the entire process of an online conference, enabling the conference to proceed efficiently and effectively.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, either from a calendar API or a database.

[0063] Step 2:

[0064] The server uses the information it retrieves to send reminders to participants, either via email or a notification system.

[0065] Step 3:

[0066] When the meeting starts, the server generates a meeting link, which is a specific URL.

[0067] Step 4:

[0068] The server notifies participants of the generated conference link via email or chat.

[0069] Step 5:

[0070] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[0071] Step 6:

[0072] During a conference, users can send requests to speak through their devices, using buttons or the chat function.

[0073] Step 7:

[0074] The device sends a request to the server, which sends a POST request to the server's API.

[0075] Step 8:

[0076] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[0077] Step 9:

[0078] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[0079] Step 10:

[0080] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[0081] Step 11:

[0082] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[0083] Step 12:

[0084] The server shares the summary with participants via email or cloud storage.

[0085] Step 13:

[0086] During a meeting, a user enters a question in the chat box and presses the send button.

[0087] Step 14:

[0088] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[0089] Step 15:

[0090] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[0091] Step 16:

[0092] The server notifies participants of action items, either via email or through a notification system.

[0093] Step 17:

[0094] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Conventional online conference systems require manual management and progress of meetings, which requires a lot of time and effort. Furthermore, there are often ineffective management of the order of comments, recording of discussions, and management of questions and answers during meetings, which reduces the efficiency of meetings. Furthermore, insufficient follow-up after the meeting can lead to delays in the implementation of action items.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating an AI moderator, receiving speech requests, and organizing the speaking order, means for receiving speech requests from terminals and adding them to a queue, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into a chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, and means for sending follow-up notifications after the end of the meeting. This automates the entire process of online meetings, enabling efficient and effective meeting conduct.

[0100] "Scheduled date and time of the meeting" refers to the specific date and time for which the meeting is scheduled.

[0101] "Participant List" means a list of all participants invited to a Conference.

[0102] An "agenda" is a list of items or topics to be discussed at a meeting.

[0103] "Means for sending reminders" refers to a function that notifies meeting participants so they don't forget about the meeting.

[0104] "Means for generating a meeting link" refers to the function for creating a URL for joining an online meeting.

[0105] "Means for activating AI moderation" refers to the function of activating artificial intelligence that automatically manages the progress of the meeting.

[0106] "Means for receiving a speech request" refers to a function for receiving a request to speak from a participant.

[0107] "Means for organizing the speech order" refers to a function for arranging received speech requests in an appropriate order.

[0108] "Means to add to queue" refers to the ability to add a request to speak to a waiting list.

[0109] "Means of real-time transcription" refers to the ability to record audio during a meeting as text in real time.

[0110] "Means of extracting a summary" refers to the ability to extract key points and important information from the entire content of a meeting.

[0111] "Means for entering questions into chat" refers to the function by which participants enter questions using the chat function.

[0112] "Means for analyzing questions and notifying participants" refers to the function of analyzing the entered questions and notifying the relevant participants at the appropriate time.

[0113] "Means for checking action items" refers to the function for checking specific action items decided in the meeting.

[0114] "Means for sending follow-up notifications" refers to a function for sending follow-up notifications to participants after the end of a conference.

[0115] This invention relates to a system that uses AI to appropriately moderate online meetings and promote communication. This system automates a series of processes, from planning the meeting to proceeding with the meeting, recording and sharing the content, managing questions and answers, and following up after the meeting. Specific embodiments of this system are described below.

[0116] 1. Planning a meeting

[0117] 1.1 Getting the scheduled date, time, attendee list, and agenda

[0118] The server uses a calendar API or database to obtain the scheduled date and time of the meeting, the participant list, and the agenda. Specifically, it could use the Google (registered trademark) Calendar API or the Microsoft (registered trademark) Outlook API. This method allows accurate acquisition of basic information about the meeting.

[0119] 1.2 Sending reminders

[0120] The server sends reminders to participants based on the information obtained, typically using an email sending API (e.g., SendGrid) or push notifications.

[0121] 2. Starting the meeting

[0122] 2.1 Generating and notifying the meeting link

[0123] When the meeting starts, the server generates a meeting link in a specific URL format and notifies the participants. The meeting link is generated using the API of a common online meeting platform (e.g., Zoom Meetings API).

[0124] 2.2 Starting AI moderation

[0125] When the meeting starts, the server launches an AI moderator to manage the meeting. Specifically, it uses an AI model (e.g., OpenAI® GPT-3®) to support the meeting.

[0126] 3. Comment Management

[0127] 3.1 Sending a request to speak

[0128] Users send requests to speak using the buttons on their devices or the chat function, and this information is sent from the device to the server.

[0129] 3.2 Receiving requests and organizing the order of speaking

[0130] The server receives the requests and organizes the speaking order. The request is added to the queue, and the AI ​​moderator announces, "The next speaker is Mr. / Ms. XX."

[0131] 4. Record and share content

[0132] 4.1 Real-time transcription

[0133] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[0134] 4.2 Abstract Extraction and Notification

[0135] The server extracts a summary of the meeting contents and shares it with participants. It uses a summary generation model (e.g., BART) to extract the main points of the meeting and notifies them via email or other means.

[0136] 5. Question and Answer Management

[0137] 5.1 Entering a Question

[0138] Users can input questions using the chat function on their devices, and this information is sent from the devices to the server.

[0139] 5.2 Question notification and response promotion

[0140] The server receives the question, and the AI ​​moderator prompts the answer at the appropriate time, notifying the user, "There is a new question. Mr. / Ms. XX, please answer."

[0141] 6. Review and follow up on action items

[0142] 6.1 Extracting Action Items

[0143] Before the end of the meeting, the server checks the action items and extracts any uncompleted tasks or action targets.

[0144] 6.2 Follow-up Notifications

[0145] The server will send periodic follow-up notifications after the meeting to help ensure action items are carried out.

[0146] Specific examples

[0147] Example 1: Send a reminder

[0148] The server retrieves meeting information scheduled for "October 15, 2023 at 2:00 PM" from the Google Calendar API and sends reminders to participants using the SendGrid API.

[0149] Example 2: Speech Management

[0150] During a meeting, when a user clicks a button on their device to send a request to speak, the server receives the request and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[0151] Example 3: Managing questions and answers

[0152] During a meeting, if a user types in chat, "More details about this topic?", the server receives the question and the AI ​​moderator notifies them, "There is a new question. Mr. Sato, please answer."

[0153] Prompt Sentence Examples

[0154] Please tell me more about how to notify "X-Person will be speaking next."

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Step 1: Get the scheduled meeting date, time, attendee list, and agenda

[0157] The server retrieves the scheduled date and time of the meeting, the attendee list, and the agenda from a calendar API (e.g., Google Calendar API) or a database. In this step, the server confirms the meeting date, attendees, and agenda based on the data retrieved through the API. The input is information from the calendar API or database, and the output is the meeting information.

[0158] Specific behavior:

[0159] The server sends a request to the Google Calendar API to retrieve the meeting information.

[0160] The server reads the participant list from a database.

[0161] Step 2: Send a reminder

[0162] The server sends reminders based on the acquired meeting information. This process uses an email sending API (e.g., SendGrid) or a push notification service to notify participants. The input is the meeting information acquired in step 1, and the output is the sent reminder notification.

[0163] Specific behavior:

[0164] The server uses the SendGrid API to send a reminder to participants saying, "We're planning a meeting for October 15th."

[0165] The server uses a push notification service to send notifications to participants' devices.

[0166] Step 3: Generate and share the meeting link

[0167] When the meeting start time arrives, the server uses the API of the online meeting platform (e.g., Zoom Meetings API) to generate a meeting link and notify participants. The input is the meeting start time and participant information, and the output is the generated meeting link and its notification.

[0168] Specific behavior:

[0169] The server uses the Zoom Meetings API to generate the meeting link.

[0170] The server will send the generated link to participants via email and push notification.

[0171] Step 4: Start the AI ​​moderator

[0172] The server starts the AI ​​moderator as soon as the meeting starts and manages the progress of the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the progress of the meeting. The input is the trigger to start the meeting, and the output is the startup status of the AI ​​moderator.

[0173] Specific behavior:

[0174] The server launches OpenAI GPT-3 at the start of the meeting and begins moderating.

[0175] Step 5: Sending and receiving requests to speak

[0176] Users send speech requests using the buttons or chat function on their devices, and the devices send this information to the server. The server receives the requests and organizes the speech order. The input is the speech request from the user, and the output is the organized speech order.

[0177] Specific behavior:

[0178] The user clicks the "Request to speak" button on the device.

[0179] The terminal sends a request to speak to the server.

[0180] The server receives the request and adds it to a queue.

[0181] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. XX."

[0182] Step 6: Real-time transcription

[0183] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is the audio data during the meeting, and the output is the transcribed text.

[0184] Specific behavior:

[0185] The server sends the audio data to the Google Cloud Speech-to-Text API, which returns it as text.

[0186] Transcribe what is being said in real time.

[0187] Step 7: Extract and notify the summary

[0188] The server extracts a summary of the meeting content and notifies participants. The summary is created using a summary generation model (e.g., BART). The input is the transcribed text, and the output is the summarized text.

[0189] Specific behavior:

[0190] The server uses the BART model to extract summaries from the transcribed content.

[0191] The server will notify participants of the summary via email.

[0192] Step 8: Manage questions and answers

[0193] Users input questions into the chat through their devices, the server receives and analyzes the questions, and the AI ​​moderator facilitates the answers. The input is the user's question, and the output is the analyzed question and answer instructions.

[0194] Specific behavior:

[0195] The user types a question into the chat box.

[0196] The terminal sends a query to the server.

[0197] The server notifies the AI ​​moderator of the question and prompts them, "There is a new question. Mr. / Ms. XX, please answer."

[0198] Step 9: Review and follow up on action items

[0199] The server checks action items before the end of the meeting, extracts uncompleted tasks and action targets, and notifies participants. It also periodically sends follow-up notifications after the meeting ends. The input is the action items recorded during the meeting, and the output is confirmation and follow-up notifications.

[0200] Specific behavior:

[0201] The server checks the task list for the meeting and extracts any uncompleted items.

[0202] The server will send follow-up notifications one week after the meeting ends.

[0203] (Application example 1)

[0204] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] Currently, maintaining efficiency and consistency in online meetings and virtual events is difficult, particularly in managing participants' opinions and smoothly managing the flow of real-time questions and answers. Furthermore, in environments with a large number of participants, such as online product briefings and campaign events held in virtual stores, managing speaking requests and questions becomes even more cumbersome, hindering effective communication.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0207] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, and means for managing speech requests and questions. This enables efficient and consistent management of the flow of speech requests and questions at online product briefings and campaign events in virtual stores, enabling effective communication.

[0208] The "scheduled date and time of the meeting" refers to the date and time when the meeting will start.

[0209] A "participant list" refers to a list of people who plan to attend a meeting or event.

[0210] An "agenda" is a list of topics or items to be discussed at a meeting or event.

[0211] "Means for sending reminders" means a function or system for sending advance notice of a meeting or event to participants.

[0212] "Conference Link" means a specific web address or invitation link through which participants can join a conference or event.

[0213] "Means of notification" refers to a communication means for conveying information such as a conference link to participants.

[0214] "AI moderation" refers to a system that uses artificial intelligence to automatically manage the progress of meetings and events.

[0215] "Request to speak" means a request by a participant for permission to speak during a meeting or event.

[0216] "Means for organizing the order of speech" refers to a system or function that accepts speech requests from participants and arranges the speech in the appropriate order.

[0217] "Means of real-time transcription" refers to functions or systems that convert statements made during meetings or events into text data in real time.

[0218] "Means to extract summaries and share them with participants" refers to a function for summarizing the contents of a meeting or event and conveying the main points to participants.

[0219] "Means for analyzing questions and notifying participants" refers to a system or function for analyzing questions entered in chat and informing the appropriate participants of their content.

[0220] "Means to review action items and notify participants" refers to the ability to review outstanding tasks and action items before the end of a meeting or event and notify participants as needed.

[0221] "Means for sending follow-up notifications" refers to a system or function for checking the progress of action items after a meeting or event and sending follow-up notifications to participants.

[0222] "Virtual store" refers to a virtual store that operates on the Internet, an online sales and service platform that does not have an actual physical location.

[0223] An "online product briefing" refers to a product briefing event held over the Internet.

[0224] "Campaign Event" means a series of online events conducted to promote a particular product or service.

[0225] "Means for managing speech requests and questions" refers to a system or function for managing speech requests and questions from participants during an event in a virtual store.

[0226] This invention relates to a system and method for smoothly conducting online product briefings and campaign events in a virtual store. This system uses AI to automate the event moderation and provides functions for managing participants' comments and smoothly managing the flow of questions and answers. The implementation of this system includes the following specific configuration and operation procedures.

[0227] Hardware and software used

[0228] server:

[0229] Role: Overall moderator, sending reminders, generating meeting links, managing speaking requests and questions, real-time transcription, extracting summaries, and following up notifications.

[0230] Examples: Google Cloud Platform, AWS (registered trademark) (Amazon Web Services)

[0231] Device:

[0232] Role: A device for participants to send requests to speak and type questions through chat.

[0233] Examples: smartphones, tablets, and PCs

[0234] Server processing

[0235] 1. Send reminders:

[0236] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda, and sends reminders to participants, allowing them to be aware of the meeting or event in advance.

[0237] Example: A server sends reminder emails to participants on the weekend about next week's event.

[0238] 2. Conference link generation and notification:

[0239] At the start time of the meeting, the server generates a meeting link and notifies participants, who can use this link to join the online event.

[0240] Example: Before the event starts, a meeting link is generated and sent to participants' email addresses.

[0241] 3. Managing requests and questions:

[0242] Participants can send requests to speak through their devices, and the server adds them to a queue and organizes them in the appropriate order.

[0243] Additionally, when a question is sent using the chat function, the server analyzes it and notifies the appropriate participant.

[0244] Example: During an event, a participant sends a request to speak, and the server organizes the order and notifies them, "Person A will speak next." Also, a question is sent via chat, and the server notifies them, "There is a new question. Person B, please answer."

[0245] 4. Real-time transcription and summary extraction:

[0246] The server transcribes speeches made during the event in real time, and a summarization algorithm is used to extract key points and share them with participants.

[0247] Example: After the event, a transcript and summary of what was said is automatically emailed to attendees.

[0248] 5. Follow-up Notification:

[0249] Before the event ends, the server checks for action items, extracts any incomplete tasks and action items, and notifies participants. After the event ends, the server periodically sends follow-up notifications to ensure that action items are carried out.

[0250] Example: After the event ends, the server notifies participants that "Uncompleted task: Create explanatory materials about the product details."

[0251] Examples of prompt statements

[0252] Implement a function send_reminder to send reminders to participants. It uses Gmail's SMTP server and takes the specified email address, subject, and body as arguments. It also requires user authentication to send the email.

[0253] Through the above process, the present invention can realize effective communication and smooth event progress within the virtual store.

[0254] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0255] Step 1:

[0256] The server obtains the scheduled date and time of the meeting, the participant list, and the agenda. This information is obtained as input from a database or calendar API. The output is the obtained scheduled date and time, participant list, and agenda data. Specifically, the server calls the API to extract the data and saves it in an internal database.

[0257] Step 2:

[0258] The server sends reminders to participants based on the information it obtains. The input is the scheduled date and time of the meeting, the participant list, and the agenda. The output is a reminder email or notification sent to participants. Specifically, the server generates the reminder content and sends the email using an SMTP server.

[0259] Step 3:

[0260] At the start time of the conference, the server generates a conference link and notifies the participants. The input is the start time and the participant list. The output is the generated conference link and an email notifying the participants. Specifically, the server generates a conference URL and sends a notification via email.

[0261] Step 4:

[0262] A user uses a terminal to send a speech request. The input is the user's request information. The output is adding the request information to the speech queue by sending it to the server. In concrete terms, the user inputs a request through the interface of the terminal, and the terminal sends the information to the server.

[0263] Step 5:

[0264] The server receives speech requests and organizes the speech order. The input is request information from the user. The output is an organized speech order list. Specifically, the server adds requests to a queue and manages the requests in the appropriate order.

[0265] Step 6:

[0266] A user inputs a question into the chat, and the terminal sends the question to the server. The input is the question entered by the user. The output is the question being sent to the server. In concrete terms, the user inputs a question into the chat interface and presses the send button.

[0267] Step 7:

[0268] The server analyzes the question and notifies the appropriate participant. The input is the question sent by the user via chat. The output is data used to analyze and notify the question. Specifically, the server uses natural language processing to analyze the question and notify the appropriate respondent.

[0269] Step 8:

[0270] The server transcribes the meeting content in real time and extracts a summary. The input is the audio data of the meeting. The output is the transcribed text data and summary data. Specifically, the server uses a speech recognition engine to convert the audio data into text and then summarizes the text using a summarization algorithm.

[0271] Step 9:

[0272] The server checks action items and notifies participants before the end of the meeting. The input is the list of action items recorded during the meeting. The output is to notify participants of incomplete action items. Specifically, the server scans the action item list and adds incomplete tasks to the notification list.

[0273] Step 10:

[0274] The server sends follow-up notifications after the meeting ends. The input is the progress of the action items at the time the meeting ends. The output is a follow-up notification email or notification. Specifically, the server checks the status of the action items and periodically sends follow-up notifications.

[0275] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0276] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and further combines it with an emotion engine that recognizes user emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[0277] System programs and their processing

[0278] 1. Planning a meeting

[0279] First, the server retrieves the scheduled meeting date and time, the attendee list, and the agenda. This information can be retrieved from a calendar API or a database. Then, based on this information, the server sends reminders to the attendees. Reminders can be sent via email or a notification system.

[0280] 2. Starting the meeting

[0281] When the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. The server then launches an AI moderator to manage the meeting.

[0282] 3. Comment Management

[0283] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[0284] 4. Record and share content

[0285] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[0286] 5. Question and Answer Management

[0287] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[0288] 6. Review and follow up on action items

[0289] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[0290] 7. Emotion Engine Functions

[0291] By incorporating an emotion engine, the server can analyze the user's emotional state in real time during a meeting. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, and speech content, and analyzes and stores these as data.

[0292] The AI ​​moderator can generate appropriate responses and comments based on this emotional data, reducing stress for participants and maintaining a good meeting atmosphere. For example, if a user is nervous, the AI ​​moderator will say, "Please relax and speak."

[0293] Specific examples

[0294] Example 1: Planning a meeting and sending reminders

[0295] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday and sends reminders to participants via email, reminding them to remember the meeting date.

[0296] Example 2: Speech management and emotion recognition

[0297] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to the queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next." At the same time, the emotion engine analyzes the user's emotional state, and if they appear tense, it will say, "Please relax and speak."

[0298] Example 3: Question and answer management and sentiment analysis

[0299] During a meeting, a user types a question into the chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Sato-san, please answer." At the same time, the emotion engine analyzes Sato-san's anxious expression and provides additional support, such as, "If the question is complicated, please speak more slowly."

[0300] In this way, the system of the present invention not only automates the entire process of online meetings, enabling efficient and effective meeting conduct, but also supports more human and interactive meetings through the introduction of an emotion engine.

[0301] The processing flow will be explained below.

[0302] Step 1:

[0303] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, which can be retrieved from a calendar API or a database.

[0304] Step 2:

[0305] The server uses the information it retrieves to send reminders to participants, which can be sent via email or a notification system.

[0306] Step 3:

[0307] When the meeting starts, the server generates a meeting link, which is a specific URL.

[0308] Step 4:

[0309] The server notifies participants of the generated conference link via email or chat.

[0310] Step 5:

[0311] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[0312] Step 6:

[0313] Users send requests to speak through their devices, using buttons or the chat function.

[0314] Step 7:

[0315] The device sends a request to the server, which sends a POST request to the server's API.

[0316] Step 8:

[0317] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[0318] Step 9:

[0319] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[0320] Step 10:

[0321] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[0322] Step 11:

[0323] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[0324] Step 12:

[0325] The server shares the summary with participants via email or cloud storage.

[0326] Step 13:

[0327] During a meeting, a user enters a question in the chat box and presses the send button.

[0328] Step 14:

[0329] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[0330] Step 15:

[0331] During the meeting, the server uses an emotion engine to analyze the user's emotional state by detecting facial expressions, tone of voice, and speech content and analyzing the data.

[0332] Step 16:

[0333] The emotion engine analyzes the user's emotional state and sends the results back to the server. The emotional data is then passed on to the AI ​​host.

[0334] Step 17:

[0335] Based on the emotional data, the AI ​​moderator generates appropriate comments, such as "Please relax and speak," and notifies the speaker in real time through voice synthesis and text display.

[0336] Step 18:

[0337] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[0338] Step 19:

[0339] The server notifies participants of action items, either via email or through a notification system.

[0340] Step 20:

[0341] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[0342] Example 2

[0343] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] Ensuring smooth online meetings and smooth communication between participants requires a lot of manual work. This includes planning meetings, sending reminders, managing comments, recording and sharing content, coordinating questions and answers, and following up after the meeting. It is also important to understand participants' emotional states and communicate appropriately. However, performing these tasks manually is labor-intensive and inefficient. Furthermore, analyzing participants' emotions in real time and responding appropriately is practically difficult. Therefore, there is a need for an automated system to solve these issues and improve the efficiency and quality of online meetings.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0346] In this invention, the server includes: means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda; means for sending reminders to participants; means for generating a meeting link at the start time of the meeting and notifying participants; means for activating an AI moderator, receiving speaking requests, and organizing the speaking order; means for transcribing the meeting content in real time; means for extracting a summary of the meeting content and sharing it with participants; means for inputting questions into a chat, analyzing the questions, and notifying participants; means for checking action items before the end of the meeting and notifying participants; means for sending follow-up notifications after the end of the meeting; and means for analyzing user emotions in real time using an emotion engine and generating appropriate responses and comments. This automates the entire process of online meetings, enabling efficient and effective meeting conduct and also realizing smoother communication by understanding participants' emotions in real time and responding appropriately.

[0347] The "scheduled date and time of the conference" refers to the specific date and time when the conference is scheduled to start.

[0348] A "participant list" is a list that includes information such as names and contact details of people who will be attending a conference.

[0349] An agenda is a list of the main topics and subjects of a meeting that is compiled in advance and used as a guideline for proceeding.

[0350] A "reminder" is an email or message that notifies participants in advance so they don't forget about the meeting schedule.

[0351] "Meeting Link" means a specific URL used by an online meeting platform that allows participants to join a meeting by clicking on the link.

[0352] "AI moderation" is a system that uses artificial intelligence to manage the progress of a meeting, automatically coordinating remarks and managing questions.

[0353] A "request to speak" is a request made by a participant who wishes to speak during a conference to inform the moderator of that request.

[0354] "Speaking order" refers to the order in which people speak in a meeting, and is adjusted to ensure smooth progress.

[0355] "Transcription" is the process or result of converting audio from a meeting into text in real time.

[0356] A "summary" is a concise summary of the main points and conclusions discussed at a meeting that is shared with all participants.

[0357] "Chat" is a real-time means of communication using text messages.

[0358] "Action items" refer to specific actions or tasks decided during a meeting.

[0359] "Follow-up notifications" are reminders sent to participants after the meeting to ensure action items are carried out.

[0360] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, speech content, etc. to recognize their emotional state and generate an appropriate response.

[0361] "Appropriate responses and comments" are statements and feedback generated by AI based on the analyzed emotional state, designed to reduce participants' stress and create a positive atmosphere in the meeting.

[0362] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. It also incorporates an emotion engine that recognizes the user's emotions. This system automates a series of processes, from planning the meeting, conducting the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[0363] First, the server retrieves the scheduled meeting date and time, the attendee list, and the agenda. This information can be retrieved from a calendar API, such as the Google Calendar API or the Microsoft Exchange API, or from an internal database. Based on this information, the server generates and sends reminders to participants via email or a notification system (e.g., Slack or MICROSOFT TEAMS®).

[0364] Next, as the meeting's start time approaches, the server generates a meeting link to be used in an online meeting platform (such as Zoom, Microsoft Teams, or GOOGLE MEET®) and notifies participants. Notifications are sent via email or chat tools. At the same time, the server launches an AI moderator program to manage the progress of the meeting.

[0365] During a meeting, users use their devices (PC, tablet, smartphone) to send requests to speak. This request is sent using the device's buttons or chat function, and the device then sends it to the server. The server receives the requests and organizes the speaking order. The AI ​​moderator notifies participants, such as "Person XX will be speaking next," helping to ensure smooth progress.

[0366] The server transcribes what is said during the meeting in real time. It uses engines such as Google Cloud Speech-to-Text, IBM Watson® Speech to Text, and Microsoft Azure® Cognitive Services for speech recognition. The server then summarizes the meeting, extracts key points, and shares them with participants via email or in a document format.

[0367] During the meeting, users can use the chat function to enter questions. The questions entered in the chat are received by the server, and the AI ​​moderator notifies them by saying, "There is a new question. Mr. / Ms. X, please answer."

[0368] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. After the meeting ends, the server periodically sends follow-up notifications to help ensure that action items are carried out. This follow-up is achieved by using reminder emails and integration with task management tools (Trello and Asana).

[0369] The system also incorporates an emotion engine. The server uses an emotion engine (such as Microsoft Azure Emotion API, Amazon Rekognition, or Google Cloud AI) to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the analysis results, the AI ​​moderator generates appropriate responses and comments to reduce user stress and maintain a good meeting atmosphere. For example, it might say to a nervous user, "Please relax and speak."

[0370] Specific examples

[0371] Meeting planning and reminders

[0372] The server calls the Google Calendar API to retrieve meeting information for the next week, and automatically sends reminders to all participants via Gmail based on the meeting date, time, and participant list.

[0373] Meeting link generation and notification

[0374] 15 minutes before the meeting starts, the server creates a new meeting using the Zoom API and generates a meeting link, which is then sent to participants via Slack.

[0375] Organizing requests to speak

[0376] The user clicks the "Request to speak" button on their PC. The device sends this request to the server, which adds it to the queue. The AI ​​moderator announces, "Tanaka will be speaking next." At the same time, the emotion engine analyzes Tanaka's facial expressions and, if she appears nervous, says, "Please relax and speak."

[0377] Managing Questions and Answers

[0378] The user types in the chat box, "When is this assignment due?" The server receives this message, and the AI ​​moderator announces, "There is a new question. Sato-san, please answer."

[0379] Prompt Sentence Examples

[0380] "How should I address a participant who seems nervous during a remote meeting?"

[0381] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0382] Step 1:

[0383] Meeting planning

[0384] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda. This is done by using the Google Calendar API, Microsoft Exchange API, and also by reading data from an internal database. Specifically, it sends a request to the API endpoint and parses the data returned as a response to retrieve the meeting information. This information is the input data, and the retrieved date and time of the meeting, the participant list, and the agenda are the output data.

[0385] Step 2:

[0386] Send a reminder

[0387] The server generates reminders based on the scheduled date and time, participant list, and agenda of the meeting. These reminders are sent to participants via email or notification system via Gmail API or Slack API. The input data is the retrieved meeting information, and the output data is the reminder sending status.

[0388] Step 3:

[0389] Generate a Conference Link

[0390] When the meeting start time approaches, the server generates a new meeting link using the Zoom API, Microsoft Teams API, or Google Meet API. It sends an appropriate request to the API and obtains the meeting link as a response. The input data is the meeting start time, and the output data is the generated meeting link.

[0391] Step 4:

[0392] Meeting link notification

[0393] The server notifies participants of the generated meeting link via email or chat tool. The Slack API is used to notify all participants of the link. The input data is the generated meeting link, and the output data is the notification sending status.

[0394] Step 5:

[0395] AI moderator activation

[0396] The server starts an AI moderator program at the same time as the start of the meeting. The moderator program manages the progress of the meeting using a generative AI model (e.g., GPT-3). The input data is the start signal for the meeting, and the output data is the start status of the AI ​​moderator program.

[0397] Step 6:

[0398] Sending a request to speak

[0399] A user uses a terminal to send a speech request, and the terminal sends this request to the server. The input data is the user's speech request, and the output data is the request status sent to the server.

[0400] Step 7:

[0401] Receiving and organizing requests to speak

[0402] The server organizes speech requests received from terminals in order, adds the requests to a queue, and selects the next user to speak. The input data is the received speech requests, and the output data is the organized speech order.

[0403] Step 8:

[0404] Speaking order notification

[0405] The server notifies the AI ​​moderator, "The next speaker is Mr. / Ms. X." The input data is the organized order of speeches, and the output data is the notification sending status.

[0406] Step 9:

[0407] Meeting transcription

[0408] The server transcribes the meeting content in real time using a speech recognition engine such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The input data is the meeting audio, and the output data is the transcribed text.

[0409] Step 10:

[0410] Summarizing and sharing meeting content

[0411] The server processes the transcribed text with a summarization algorithm to extract key points, and shares the extracted summary with participants via email or document format. The input data is the transcribed text, and the output data is the summarized text and the sharing status.

[0412] Step 11:

[0413] Receiving and parsing questions

[0414] The user enters a question using the chat function. The server receives the question, and the AI ​​moderator analyzes the question and notifies the person responsible for answering it. The input data is the question entered in the chat, and the output data is the analysis result and notification status.

[0415] Step 12:

[0416] Action item confirmation and notifications

[0417] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. The input data is the meeting summary and task list, and the output data is the notification status.

[0418] Step 13:

[0419] Sending follow-up notifications

[0420] Even after the meeting ends, the server periodically sends follow-up notifications. It uses reminder emails and task management tools to ensure that action items are carried out. The input data is a list of action items, and the output data is the sending status of the follow-up notifications.

[0421] Step 14:

[0422] Emotion engine analysis and response generation

[0423] The server uses an emotion engine to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the results of this analysis, the AI ​​moderator generates appropriate responses and comments. For example, if the server detects that the user is nervous, it generates a message such as "Please relax and speak." The input data is the user's emotional state, and the output data is the response message.

[0424] (Application example 2)

[0425] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0426] With conventional online conference systems, planning, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting are all done manually, making it difficult to manage efficiently. Furthermore, the meeting cannot proceed taking into account the emotional state of the participants, resulting in insufficient communication. This results in issues such as meetings not proceeding smoothly and participants' opinions not being fully elicited.

[0427] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the scheduled date and time of the meeting, the participant list, and the agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating AI moderation, receiving speech requests, and organizing the speaking order, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into the chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, means for sending follow-up notifications after the end of the meeting, and means for collecting and analyzing emotional data and generating appropriate responses and comments according to the emotional states of the participants. This not only automates the entire process of online meetings, but also enables appropriate responses according to the emotional states of the participants, thereby achieving efficient and effective meeting progress.

[0428] The "scheduled date and time of the meeting" refers to the specific date and time when the meeting will start.

[0429] The "participant list" indicates a list of users who are scheduled to participate in the conference.

[0430] An "agenda" is a list of specific topics or issues to be discussed at a meeting.

[0431] "Means for sending reminders" refers to a method or system for notifying users of upcoming meetings.

[0432] "Conference link" refers to the URL and connection information required for a user to participate in a conference.

[0433] "AI moderation" refers to a system that uses artificial intelligence to manage the progress of online meetings and call on speakers at appropriate times.

[0434] "Means for receiving requests to speak and organizing the order in which they speak" refers to a method or system for receiving requests to speak from participants and managing them in an appropriate order.

[0435] "Means for transcribing meeting content in real time" refers to technology that instantly converts audio during a meeting into text.

[0436] "Means for extracting a summary of the meeting content and sharing it with participants" refers to a method or system for summarizing the important points and main points from the meeting and communicating them to participants.

[0437] "Means of entering questions into chat, analyzing the questions, and notifying participants" refers to a method of receiving text questions from participants, understanding their content, and notifying participants at an appropriate time.

[0438] An "action item" refers to a specific task or work decided during a meeting.

[0439] "Means for sending follow-up notifications" refers to a method or system for checking whether action items are being properly carried out after the meeting and sending reminders as necessary.

[0440] "Emotion data" refers to information about the user's emotional state that can be inferred from facial expressions, tone of voice, and the like.

[0441] "Means for generating appropriate responses and comments" refers to methods and systems for providing necessary support and encouragement to participants based on emotional data.

[0442] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and combines it with an emotion engine that recognizes the user's emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[0443] System programs and their processing

[0444] Hardware and Software

[0445] Hardware: Servers, devices (smartphones, PCs, etc.)

[0446] Software: SQLite (database), smtplib (Python library), email.mime (Python library), speech recognition engine (e.g., Google Speech-to-Text API), sentiment analysis engine (e.g., Amazon Comprehend)

[0447] Meeting planning

[0448] The server retrieves the scheduled meeting date and time, the participant list and the agenda from the database and sends reminders to participants, which are sent via email.

[0449] Specific examples

[0450] For a meeting scheduled for next Monday, the server retrieves the scheduled date and time, the attendee list, and the agenda from the database, and sends email reminders to the attendees, helping them remember to plan for the meeting.

[0451] Example prompt sentence:

[0452] "Get the date, time, participant list, and agenda for your next seminar and send reminder emails to each participant."

[0453] Starting a meeting

[0454] When the meeting starts, the server generates a meeting link and notifies the participants, who can then join the meeting using the generated URL.

[0455] Speech Management

[0456] Users send requests to speak through their devices, which are received by the server and sorted in the appropriate order. The AI ​​moderator announces the order of speaking and helps ensure the meeting proceeds smoothly.

[0457] Specific examples

[0458] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[0459] Record and share meeting content

[0460] The server transcribes what is said during the meeting in real time, and a summary is extracted using a summarization algorithm and shared with participants, allowing all participants to accurately understand the content of the meeting.

[0461] Managing Questions and Answers

[0462] During the meeting, users can enter questions into the chat through their devices. The questions are received by the server, and the AI ​​moderator notifies them at the appropriate time. This function allows questions and answers to be exchanged smoothly.

[0463] Specific examples

[0464] During a meeting, a user enters a question via chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Mr. Sato, please answer."

[0465] Example prompt sentence:

[0466] "Receive chat questions and notify us at the right time"

[0467] Review and follow up on action items

[0468] The server checks for action items and notifies participants before the end of the meeting, and also sends follow-up notifications after the meeting to ensure that action items are carried out.

[0469] Emotion Engine Functions

[0470] By incorporating an emotion engine, the server analyzes the user's emotional state in real time during the meeting, allowing the AI ​​moderator to reduce participants' stress and maintain a good meeting atmosphere.

[0471] Specific examples

[0472] During a meeting, the emotion engine analyzes the user's level of tension from their facial expressions and tone of voice, and the AI ​​moderator encourages them to "relax and speak."

[0473] As described above, by using the above-mentioned hardware and software, the online conference system of the present invention can automate the entire conference process and respond appropriately to the emotional state of the participants, thereby realizing efficient and effective conference proceedings.

[0474] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0475] Step 1:

[0476] The server retrieves the scheduled date and time, participant list, and agenda of the meeting from the database. It receives a meeting ID as input and searches the database for the corresponding scheduled date and time, participant list, and agenda of the meeting and outputs it. This data is used as information required for subsequent processing.

[0477] Step 2:

[0478] The server sends reminders to participants based on the information it has obtained. It receives the participant list, scheduled date and time, and agenda as input, creates and sends an email, and receives a confirmation that the reminder was sent as output. The reminder helps participants remember that the meeting is happening.

[0479] Step 3:

[0480] When the meeting start time arrives, the server generates a meeting link and notifies the participants. It receives the meeting ID as input and generates a meeting link (URL). The generated URL is sent as output to the participants via email. This prepares the participants to join the meeting.

[0481] Step 4:

[0482] The server starts the AI ​​moderator, receives speaking requests, and organizes the speaking order. It receives speaking requests as input and organizes the speaking order. It adds the requests to the queue in the appropriate order. It outputs organized speaking order information. This allows the meeting to proceed smoothly.

[0483] Step 5:

[0484] The server transcribes what is said during the meeting in real time. It receives the voice data from the meeting as input and converts it into text data using a speech recognition engine. It obtains the text data generated in real time as output. This records the contents of the meeting.

[0485] Step 6:

[0486] The server extracts a summary of the meeting content and shares it with participants. It receives the transcribed text data as input and uses a summarization algorithm to extract key points. It generates the extracted summary as output and shares it with participants, thereby conveying the important points of the meeting to participants.

[0487] Step 7:

[0488] Users input questions through chat during a meeting. The text-based question is received as input and sent to the server. The server analyzes the question and notifies the participants. The analyzed question is obtained as output, and the AI ​​moderator makes notifications based on that. This allows for a smooth exchange of questions and answers.

[0489] Step 8:

[0490] Before the end of the meeting, the server confirms the action items and notifies the participants. It receives as input the action items decided during the meeting and notifies the participants. It outputs a confirmation that the notification was sent, ensuring that the meeting decisions are carried out.

[0491] Step 9:

[0492] After the meeting ends, the server sends a follow-up notification. It receives as input the status of the post-meeting action item, creates and sends a reminder, and as output, provides confirmation that the follow-up notification was sent, ensuring that the action item is followed up appropriately.

[0493] Step 10:

[0494] The server collects and analyzes emotional data, generating appropriate responses and comments according to the participants' emotional state. It receives emotional data such as the user's facial expressions and tone of voice as input, and analyzes it using an emotion analysis engine. It generates appropriate responses and comments based on the analysis results as output, and notifies the participants. This reduces stress for participants and maintains a good meeting atmosphere.

[0495] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0496] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0497] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0498] [Second embodiment]

[0499] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0500] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0501] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0502] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0503] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0504] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0505] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0506] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0507] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0508] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0509] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0510] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0511] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting.

[0512] System programs and their processing

[0513] 1. Planning a meeting

[0514] First, the server retrieves the scheduled meeting date and time, the list of attendees, and the agenda. This information can be retrieved from a calendar API or a database. Based on this information, the server sends reminders to attendees. These reminders can be sent via email or a notification system.

[0515] 2. Starting the meeting

[0516] Next, when the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. When the meeting starts, the server launches an AI moderator to manage the progress of the meeting.

[0517] 3. Comment Management

[0518] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[0519] 4. Record and share content

[0520] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[0521] 5. Question and Answer Management

[0522] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[0523] 6. Review and follow up on action items

[0524] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[0525] Specific examples

[0526] Example 1: Planning a meeting

[0527] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday. With this information, the server sends reminders to participants via email, ensuring they don't forget the meeting date.

[0528] Example 2: Speech Management

[0529] During a meeting, a user sends a request to speak from their device. The device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will speak next."

[0530] Example 3: Managing questions and answers

[0531] During a meeting, a user can enter a question via chat. The server receives the question, and the AI ​​moderator responds by saying, "A new question has arrived. Mr. Sato, please answer." This function ensures a smooth flow of questions and answers.

[0532] In this way, the system of the present invention can automate the entire process of an online conference, enabling the conference to proceed efficiently and effectively.

[0533] The processing flow will be explained below.

[0534] Step 1:

[0535] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, either from a calendar API or a database.

[0536] Step 2:

[0537] The server uses the information it retrieves to send reminders to participants, either via email or a notification system.

[0538] Step 3:

[0539] When the meeting starts, the server generates a meeting link, which is a specific URL.

[0540] Step 4:

[0541] The server notifies participants of the generated conference link via email or chat.

[0542] Step 5:

[0543] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[0544] Step 6:

[0545] During a conference, users can send requests to speak through their devices, using buttons or the chat function.

[0546] Step 7:

[0547] The device sends a request to the server, which sends a POST request to the server's API.

[0548] Step 8:

[0549] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[0550] Step 9:

[0551] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[0552] Step 10:

[0553] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[0554] Step 11:

[0555] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[0556] Step 12:

[0557] The server shares the summary with participants via email or cloud storage.

[0558] Step 13:

[0559] During a meeting, a user enters a question in the chat box and presses the send button.

[0560] Step 14:

[0561] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[0562] Step 15:

[0563] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[0564] Step 16:

[0565] The server notifies participants of action items, either via email or through a notification system.

[0566] Step 17:

[0567] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[0568] Example 1

[0569] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Conventional online conference systems require manual management and progress of meetings, which requires a lot of time and effort. Furthermore, there are often ineffective management of the order of comments, recording of discussions, and management of questions and answers during meetings, which reduces the efficiency of meetings. Furthermore, insufficient follow-up after the meeting can lead to delays in the implementation of action items.

[0571] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0572] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating an AI moderator, receiving speech requests, and organizing the speaking order, means for receiving speech requests from terminals and adding them to a queue, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into a chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, and means for sending follow-up notifications after the end of the meeting. This automates the entire process of online meetings, enabling efficient and effective meeting conduct.

[0573] "Scheduled date and time of the meeting" refers to the specific date and time for which the meeting is scheduled.

[0574] "Participant List" means a list of all participants invited to a Conference.

[0575] An "agenda" is a list of items or topics to be discussed at a meeting.

[0576] "Means for sending reminders" refers to a function that notifies meeting participants so they don't forget about the meeting.

[0577] "Means for generating a meeting link" refers to the function for creating a URL for joining an online meeting.

[0578] "Means for activating AI moderation" refers to the function of activating artificial intelligence that automatically manages the progress of the meeting.

[0579] "Means for receiving a speech request" refers to a function for receiving a request to speak from a participant.

[0580] "Means for organizing the speech order" refers to a function for arranging received speech requests in an appropriate order.

[0581] "Means to add to queue" refers to the ability to add a request to speak to a waiting list.

[0582] "Means of real-time transcription" refers to the ability to record audio during a meeting as text in real time.

[0583] "Means of extracting a summary" refers to the ability to extract key points and important information from the entire content of a meeting.

[0584] "Means for entering questions into chat" refers to the function by which participants enter questions using the chat function.

[0585] "Means for analyzing questions and notifying participants" refers to the function of analyzing the entered questions and notifying the relevant participants at the appropriate time.

[0586] "Means for checking action items" refers to the function for checking specific action items decided in the meeting.

[0587] "Means for sending follow-up notifications" refers to a function for sending follow-up notifications to participants after the end of a conference.

[0588] This invention relates to a system that uses AI to appropriately moderate online meetings and promote communication. This system automates a series of processes, from planning the meeting to proceeding with the meeting, recording and sharing the content, managing questions and answers, and following up after the meeting. Specific embodiments of this system are described below.

[0589] 1. Planning a meeting

[0590] 1.1 Getting the scheduled date, time, attendee list, and agenda

[0591] The server uses a calendar API or database to retrieve the scheduled date and time of the meeting, the attendee list, and the agenda. Specifically, it could use the Google Calendar API or Microsoft Outlook API. This method allows accurate acquisition of basic meeting information.

[0592] 1.2 Sending reminders

[0593] The server sends reminders to participants based on the information obtained, typically using an email sending API (e.g., SendGrid) or push notifications.

[0594] 2. Starting the meeting

[0595] 2.1 Generating and notifying the meeting link

[0596] When the meeting starts, the server generates a meeting link in a specific URL format and notifies the participants. The meeting link is generated using the API of a common online meeting platform (e.g., Zoom Meetings API).

[0597] 2.2 Starting AI moderation

[0598] When the meeting starts, the server launches an AI moderator to manage the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the meeting.

[0599] 3. Comment Management

[0600] 3.1 Sending a request to speak

[0601] Users send requests to speak using the buttons on their devices or the chat function, and this information is sent from the device to the server.

[0602] 3.2 Receiving requests and organizing the order of speaking

[0603] The server receives the requests and organizes the speaking order. The request is added to the queue, and the AI ​​moderator announces, "The next speaker is Mr. / Ms. XX."

[0604] 4. Record and share content

[0605] 4.1 Real-time transcription

[0606] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[0607] 4.2 Abstract Extraction and Notification

[0608] The server extracts a summary of the meeting contents and shares it with participants. It uses a summary generation model (e.g., BART) to extract the main points of the meeting and notifies them via email or other means.

[0609] 5. Question and Answer Management

[0610] 5.1 Entering a Question

[0611] Users can input questions using the chat function on their devices, and this information is sent from the devices to the server.

[0612] 5.2 Question notification and response promotion

[0613] The server receives the question, and the AI ​​moderator prompts the answer at the appropriate time, notifying the user, "There is a new question. Mr. / Ms. XX, please answer."

[0614] 6. Review and follow up on action items

[0615] 6.1 Extracting Action Items

[0616] Before the end of the meeting, the server checks the action items and extracts any uncompleted tasks or action targets.

[0617] 6.2 Follow-up Notifications

[0618] The server will send periodic follow-up notifications after the meeting to help ensure action items are carried out.

[0619] Specific examples

[0620] Example 1: Send a reminder

[0621] The server retrieves meeting information scheduled for "October 15, 2023 at 2:00 PM" from the Google Calendar API and sends reminders to participants using the SendGrid API.

[0622] Example 2: Speech Management

[0623] During a meeting, when a user clicks a button on their device to send a request to speak, the server receives the request and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[0624] Example 3: Managing questions and answers

[0625] During a meeting, if a user types in chat, "More details about this topic?", the server receives the question and the AI ​​moderator notifies them, "There is a new question. Mr. Sato, please answer."

[0626] Prompt Sentence Examples

[0627] Please tell me more about how to notify "X-Person will be speaking next."

[0628] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0629] Step 1: Get the scheduled meeting date, time, attendee list, and agenda

[0630] The server retrieves the scheduled date and time of the meeting, the attendee list, and the agenda from a calendar API (e.g., Google Calendar API) or a database. In this step, the server confirms the meeting date, attendees, and agenda based on the data retrieved through the API. The input is information from the calendar API or database, and the output is the meeting information.

[0631] Specific behavior:

[0632] The server sends a request to the Google Calendar API to retrieve the meeting information.

[0633] The server reads the participant list from a database.

[0634] Step 2: Send a reminder

[0635] The server sends reminders based on the acquired meeting information. This process uses an email sending API (e.g., SendGrid) or a push notification service to notify participants. The input is the meeting information acquired in step 1, and the output is the sent reminder notification.

[0636] Specific behavior:

[0637] The server uses the SendGrid API to send a reminder to participants saying, "We're planning a meeting for October 15th."

[0638] The server uses a push notification service to send notifications to participants' devices.

[0639] Step 3: Generate and share the meeting link

[0640] When the meeting start time arrives, the server uses the API of the online meeting platform (e.g., Zoom Meetings API) to generate a meeting link and notify participants. The input is the meeting start time and participant information, and the output is the generated meeting link and its notification.

[0641] Specific behavior:

[0642] The server uses the Zoom Meetings API to generate the meeting link.

[0643] The server will send the generated link to participants via email and push notification.

[0644] Step 4: Start the AI ​​moderator

[0645] The server starts the AI ​​moderator as soon as the meeting starts and manages the progress of the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the progress of the meeting. The input is the trigger to start the meeting, and the output is the startup status of the AI ​​moderator.

[0646] Specific behavior:

[0647] The server launches OpenAI GPT-3 at the start of the meeting and begins moderating.

[0648] Step 5: Sending and receiving requests to speak

[0649] Users send speech requests using the buttons or chat function on their devices, and the devices send this information to the server. The server receives the requests and organizes the speech order. The input is the speech request from the user, and the output is the organized speech order.

[0650] Specific behavior:

[0651] The user clicks the "Request to speak" button on the device.

[0652] The terminal sends a request to speak to the server.

[0653] The server receives the request and adds it to a queue.

[0654] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. XX."

[0655] Step 6: Real-time transcription

[0656] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is the audio data during the meeting, and the output is the transcribed text.

[0657] Specific behavior:

[0658] The server sends the audio data to the Google Cloud Speech-to-Text API, which returns it as text.

[0659] Transcribe what is being said in real time.

[0660] Step 7: Extract and notify the summary

[0661] The server extracts a summary of the meeting content and notifies participants. The summary is created using a summary generation model (e.g., BART). The input is the transcribed text, and the output is the summarized text.

[0662] Specific behavior:

[0663] The server uses the BART model to extract summaries from the transcribed content.

[0664] The server will notify participants of the summary via email.

[0665] Step 8: Manage questions and answers

[0666] Users input questions into the chat through their devices, the server receives and analyzes the questions, and the AI ​​moderator facilitates the answers. The input is the user's question, and the output is the analyzed question and answer instructions.

[0667] Specific behavior:

[0668] The user types a question into the chat box.

[0669] The terminal sends a query to the server.

[0670] The server notifies the AI ​​moderator of the question and prompts them, "There is a new question. Mr. / Ms. XX, please answer."

[0671] Step 9: Review and follow up on action items

[0672] The server checks action items before the end of the meeting, extracts uncompleted tasks and action targets, and notifies participants. It also periodically sends follow-up notifications after the meeting ends. The input is the action items recorded during the meeting, and the output is confirmation and follow-up notifications.

[0673] Specific behavior:

[0674] The server checks the task list for the meeting and extracts any uncompleted items.

[0675] The server will send follow-up notifications one week after the meeting ends.

[0676] (Application example 1)

[0677] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] Currently, maintaining efficiency and consistency in online meetings and virtual events is difficult, particularly in managing participants' opinions and smoothly managing the flow of real-time questions and answers. Furthermore, in environments with a large number of participants, such as online product briefings and campaign events held in virtual stores, managing speaking requests and questions becomes even more cumbersome, hindering effective communication.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0680] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, and means for managing speech requests and questions. This enables efficient and consistent management of the flow of speech requests and questions at online product briefings and campaign events in virtual stores, enabling effective communication.

[0681] The "scheduled date and time of the meeting" refers to the date and time when the meeting will start.

[0682] A "participant list" refers to a list of people who plan to attend a meeting or event.

[0683] An "agenda" is a list of topics or items to be discussed at a meeting or event.

[0684] "Means for sending reminders" means a function or system for sending advance notice of a meeting or event to participants.

[0685] "Conference Link" means a specific web address or invitation link through which participants can join a conference or event.

[0686] "Means of notification" refers to a communication means for conveying information such as a conference link to participants.

[0687] "AI moderation" refers to a system that uses artificial intelligence to automatically manage the progress of meetings and events.

[0688] "Request to speak" means a request by a participant for permission to speak during a meeting or event.

[0689] "Means for organizing the order of speech" refers to a system or function that accepts speech requests from participants and arranges the speech in the appropriate order.

[0690] "Means of real-time transcription" refers to functions or systems that convert statements made during meetings or events into text data in real time.

[0691] "Means to extract summaries and share them with participants" refers to a function for summarizing the contents of a meeting or event and conveying the main points to participants.

[0692] "Means for analyzing questions and notifying participants" refers to a system or function for analyzing questions entered in chat and informing the appropriate participants of their content.

[0693] "Means to review action items and notify participants" refers to the ability to review outstanding tasks and action items before the end of a meeting or event and notify participants as needed.

[0694] "Means for sending follow-up notifications" refers to a system or function for checking the progress of action items after a meeting or event and sending follow-up notifications to participants.

[0695] "Virtual store" refers to a virtual store that operates on the Internet, an online sales and service platform that does not have an actual physical location.

[0696] An "online product briefing" refers to a product briefing event held over the Internet.

[0697] "Campaign Event" means a series of online events conducted to promote a particular product or service.

[0698] "Means for managing speech requests and questions" refers to a system or function for managing speech requests and questions from participants during an event in a virtual store.

[0699] This invention relates to a system and method for smoothly conducting online product briefings and campaign events in a virtual store. This system uses AI to automate the event moderation and provides functions for managing participants' comments and smoothly managing the flow of questions and answers. The implementation of this system includes the following specific configuration and operation procedures.

[0700] Hardware and software used

[0701] server:

[0702] Role: Overall moderator, sending reminders, generating meeting links, managing speaking requests and questions, real-time transcription, extracting summaries, and following up notifications.

[0703] Examples: Google Cloud Platform, AWS (Amazon Web Services)

[0704] Device:

[0705] Role: A device for participants to send requests to speak and type questions through chat.

[0706] Examples: smartphones, tablets, and PCs

[0707] Server processing

[0708] 1. Send reminders:

[0709] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda, and sends reminders to participants, allowing them to be aware of the meeting or event in advance.

[0710] Example: A server sends reminder emails to participants on the weekend about next week's event.

[0711] 2. Conference link generation and notification:

[0712] At the start time of the meeting, the server generates a meeting link and notifies participants, who can use this link to join the online event.

[0713] Example: Before the event starts, a meeting link is generated and sent to participants' email addresses.

[0714] 3. Managing requests and questions:

[0715] Participants can send requests to speak through their devices, and the server adds them to a queue and organizes them in the appropriate order.

[0716] Additionally, when a question is sent using the chat function, the server analyzes it and notifies the appropriate participant.

[0717] Example: During an event, a participant sends a request to speak, and the server organizes the order and notifies them, "Person A will speak next." Also, a question is sent via chat, and the server notifies them, "There is a new question. Person B, please answer."

[0718] 4. Real-time transcription and summary extraction:

[0719] The server transcribes speeches made during the event in real time, and a summarization algorithm is used to extract key points and share them with participants.

[0720] Example: After the event, a transcript and summary of what was said is automatically emailed to attendees.

[0721] 5. Follow-up Notification:

[0722] Before the event ends, the server checks for action items, extracts any incomplete tasks and action items, and notifies participants. After the event ends, the server periodically sends follow-up notifications to ensure that action items are carried out.

[0723] Example: After the event ends, the server notifies participants that "Uncompleted task: Create explanatory materials about the product details."

[0724] Examples of prompt statements

[0725] Implement a function send_reminder to send reminders to participants. It uses Gmail's SMTP server and takes the specified email address, subject, and body as arguments. It also requires user authentication to send the email.

[0726] Through the above process, the present invention can realize effective communication and smooth event progress within the virtual store.

[0727] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0728] Step 1:

[0729] The server obtains the scheduled date and time of the meeting, the participant list, and the agenda. This information is obtained as input from a database or calendar API. The output is the obtained scheduled date and time, participant list, and agenda data. Specifically, the server calls the API to extract the data and saves it in an internal database.

[0730] Step 2:

[0731] The server sends reminders to participants based on the information it obtains. The input is the scheduled date and time of the meeting, the participant list, and the agenda. The output is a reminder email or notification sent to participants. Specifically, the server generates the reminder content and sends the email using an SMTP server.

[0732] Step 3:

[0733] At the start time of the conference, the server generates a conference link and notifies the participants. The input is the start time and the participant list. The output is the generated conference link and an email notifying the participants. Specifically, the server generates a conference URL and sends a notification via email.

[0734] Step 4:

[0735] A user uses a terminal to send a speech request. The input is the user's request information. The output is adding the request information to the speech queue by sending it to the server. In concrete terms, the user inputs a request through the interface of the terminal, and the terminal sends the information to the server.

[0736] Step 5:

[0737] The server receives speech requests and organizes the speech order. The input is request information from the user. The output is an organized speech order list. Specifically, the server adds requests to a queue and manages the requests in the appropriate order.

[0738] Step 6:

[0739] A user inputs a question into the chat, and the terminal sends the question to the server. The input is the question entered by the user. The output is the question being sent to the server. In concrete terms, the user inputs a question into the chat interface and presses the send button.

[0740] Step 7:

[0741] The server analyzes the question and notifies the appropriate participant. The input is the question sent by the user via chat. The output is data used to analyze and notify the question. Specifically, the server uses natural language processing to analyze the question and notify the appropriate respondent.

[0742] Step 8:

[0743] The server transcribes the meeting content in real time and extracts a summary. The input is the audio data of the meeting. The output is the transcribed text data and summary data. Specifically, the server uses a speech recognition engine to convert the audio data into text and then summarizes the text using a summarization algorithm.

[0744] Step 9:

[0745] The server checks action items and notifies participants before the end of the meeting. The input is the list of action items recorded during the meeting. The output is to notify participants of incomplete action items. Specifically, the server scans the action item list and adds incomplete tasks to the notification list.

[0746] Step 10:

[0747] The server sends follow-up notifications after the meeting ends. The input is the progress of the action items at the time the meeting ends. The output is a follow-up notification email or notification. Specifically, the server checks the status of the action items and periodically sends follow-up notifications.

[0748] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0749] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and further combines it with an emotion engine that recognizes user emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[0750] System programs and their processing

[0751] 1. Planning a meeting

[0752] First, the server retrieves the scheduled meeting date and time, the attendee list, and the agenda. This information can be retrieved from a calendar API or a database. Then, based on this information, the server sends reminders to the attendees. Reminders can be sent via email or a notification system.

[0753] 2. Starting the meeting

[0754] When the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. The server then launches an AI moderator to manage the meeting.

[0755] 3. Comment Management

[0756] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[0757] 4. Record and share content

[0758] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[0759] 5. Question and Answer Management

[0760] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[0761] 6. Review and follow up on action items

[0762] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[0763] 7. Emotion Engine Functions

[0764] By incorporating an emotion engine, the server can analyze the user's emotional state in real time during a meeting. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, and speech content, and analyzes and stores these as data.

[0765] The AI ​​moderator can generate appropriate responses and comments based on this emotional data, reducing stress for participants and maintaining a good meeting atmosphere. For example, if a user is nervous, the AI ​​moderator will say, "Please relax and speak."

[0766] Specific examples

[0767] Example 1: Planning a meeting and sending reminders

[0768] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday and sends reminders to participants via email, reminding them to remember the meeting date.

[0769] Example 2: Speech management and emotion recognition

[0770] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to the queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next." At the same time, the emotion engine analyzes the user's emotional state, and if they appear tense, it will say, "Please relax and speak."

[0771] Example 3: Question and answer management and sentiment analysis

[0772] During a meeting, a user types a question into the chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Sato-san, please answer." At the same time, the emotion engine analyzes Sato-san's anxious expression and provides additional support, such as, "If the question is complicated, please speak more slowly."

[0773] In this way, the system of the present invention not only automates the entire process of online meetings, enabling efficient and effective meeting conduct, but also supports more human and interactive meetings through the introduction of an emotion engine.

[0774] The processing flow will be explained below.

[0775] Step 1:

[0776] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, which can be retrieved from a calendar API or a database.

[0777] Step 2:

[0778] The server uses the information it retrieves to send reminders to participants, which can be sent via email or a notification system.

[0779] Step 3:

[0780] When the meeting starts, the server generates a meeting link, which is a specific URL.

[0781] Step 4:

[0782] The server notifies participants of the generated conference link via email or chat.

[0783] Step 5:

[0784] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[0785] Step 6:

[0786] Users send requests to speak through their devices, using buttons or the chat function.

[0787] Step 7:

[0788] The device sends a request to the server, which sends a POST request to the server's API.

[0789] Step 8:

[0790] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[0791] Step 9:

[0792] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[0793] Step 10:

[0794] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[0795] Step 11:

[0796] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[0797] Step 12:

[0798] The server shares the summary with participants via email or cloud storage.

[0799] Step 13:

[0800] During a meeting, a user enters a question in the chat box and presses the send button.

[0801] Step 14:

[0802] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[0803] Step 15:

[0804] During the meeting, the server uses an emotion engine to analyze the user's emotional state by detecting facial expressions, tone of voice, and speech content and analyzing the data.

[0805] Step 16:

[0806] The emotion engine analyzes the user's emotional state and sends the results back to the server. The emotional data is then passed on to the AI ​​host.

[0807] Step 17:

[0808] Based on the emotional data, the AI ​​moderator generates appropriate comments, such as "Please relax and speak," and notifies the speaker in real time through voice synthesis and text display.

[0809] Step 18:

[0810] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[0811] Step 19:

[0812] The server notifies participants of action items, either via email or through a notification system.

[0813] Step 20:

[0814] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[0815] Example 2

[0816] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0817] Ensuring smooth online meetings and smooth communication between participants requires a lot of manual work. This includes planning meetings, sending reminders, managing comments, recording and sharing content, coordinating questions and answers, and following up after the meeting. It is also important to understand participants' emotional states and communicate appropriately. However, performing these tasks manually is labor-intensive and inefficient. Furthermore, analyzing participants' emotions in real time and responding appropriately is practically difficult. Therefore, there is a need for an automated system to solve these issues and improve the efficiency and quality of online meetings.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0819] In this invention, the server includes: means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda; means for sending reminders to participants; means for generating a meeting link at the start time of the meeting and notifying participants; means for activating an AI moderator, receiving speaking requests, and organizing the speaking order; means for transcribing the meeting content in real time; means for extracting a summary of the meeting content and sharing it with participants; means for inputting questions into a chat, analyzing the questions, and notifying participants; means for checking action items before the end of the meeting and notifying participants; means for sending follow-up notifications after the end of the meeting; and means for analyzing user emotions in real time using an emotion engine and generating appropriate responses and comments. This automates the entire process of online meetings, enabling efficient and effective meeting conduct and also realizing smoother communication by understanding participants' emotions in real time and responding appropriately.

[0820] The "scheduled date and time of the conference" refers to the specific date and time when the conference is scheduled to start.

[0821] A "participant list" is a list that includes information such as names and contact details of people who will be attending a conference.

[0822] An agenda is a list of the main topics and subjects of a meeting that is compiled in advance and used as a guideline for proceeding.

[0823] A "reminder" is an email or message that notifies participants in advance so they don't forget about the meeting schedule.

[0824] "Meeting Link" means a specific URL used by an online meeting platform that allows participants to join a meeting by clicking on the link.

[0825] "AI moderation" is a system that uses artificial intelligence to manage the progress of a meeting, automatically coordinating remarks and managing questions.

[0826] A "request to speak" is a request made by a participant who wishes to speak during a conference to inform the moderator of that request.

[0827] "Speaking order" refers to the order in which people speak in a meeting, and is adjusted to ensure smooth progress.

[0828] "Transcription" is the process or result of converting audio from a meeting into text in real time.

[0829] A "summary" is a concise summary of the main points and conclusions discussed at a meeting that is shared with all participants.

[0830] "Chat" is a real-time means of communication using text messages.

[0831] "Action items" refer to specific actions or tasks decided during a meeting.

[0832] "Follow-up notifications" are reminders sent to participants after the meeting to ensure action items are carried out.

[0833] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, speech content, etc. to recognize their emotional state and generate an appropriate response.

[0834] "Appropriate responses and comments" are statements and feedback generated by AI based on the analyzed emotional state, designed to reduce participants' stress and create a positive atmosphere in the meeting.

[0835] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. It also incorporates an emotion engine that recognizes the user's emotions. This system automates a series of processes, from planning the meeting, conducting the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[0836] First, the server retrieves the scheduled meeting date and time, the list of participants, and the agenda. This information can be retrieved from a calendar API, such as the Google Calendar API or the Microsoft Exchange API, or from an internal database. Based on this information, the server generates and sends reminders to participants via email or a notification system (for example, Slack or Microsoft Teams).

[0837] Next, when the meeting's start time approaches, the server generates a meeting link to be used in an online meeting platform (such as Zoom, Microsoft Teams, or Google Meet) and notifies participants via email or chat. At the same time, the server launches an AI moderator program to manage the meeting.

[0838] During a meeting, users use their devices (PC, tablet, smartphone) to send requests to speak. This request is sent using the device's buttons or chat function, and the device then sends it to the server. The server receives the requests and organizes the speaking order. The AI ​​moderator notifies participants, such as "Person XX will be speaking next," helping to ensure smooth progress.

[0839] The server transcribes what is said during the meeting in real time using engines such as Google Cloud Speech-to-Text, IBM Watson Speech to Text, and Microsoft Azure Cognitive Services for speech recognition. The server then summarizes the meeting, extracts key points, and shares them with participants via email or in a document format.

[0840] During the meeting, users can use the chat function to enter questions. The questions entered in the chat are received by the server, and the AI ​​moderator notifies them by saying, "There is a new question. Mr. / Ms. X, please answer."

[0841] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. After the meeting ends, the server periodically sends follow-up notifications to help ensure that action items are carried out. This follow-up is achieved by using reminder emails and integration with task management tools (Trello and Asana).

[0842] The system also incorporates an emotion engine. The server uses an emotion engine (such as Microsoft Azure Emotion API, Amazon Rekognition, or Google Cloud AI) to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the analysis results, the AI ​​moderator generates appropriate responses and comments to reduce user stress and maintain a good meeting atmosphere. For example, it might say to a nervous user, "Please relax and speak."

[0843] Specific examples

[0844] Meeting planning and reminders

[0845] The server calls the Google Calendar API to retrieve meeting information for the next week, and automatically sends reminders to all participants via Gmail based on the meeting date, time, and participant list.

[0846] Meeting link generation and notification

[0847] 15 minutes before the meeting starts, the server creates a new meeting using the Zoom API and generates a meeting link, which is then sent to participants via Slack.

[0848] Organizing requests to speak

[0849] The user clicks the "Request to speak" button on their PC. The device sends this request to the server, which adds it to the queue. The AI ​​moderator announces, "Tanaka will be speaking next." At the same time, the emotion engine analyzes Tanaka's facial expressions and, if she appears nervous, says, "Please relax and speak."

[0850] Managing Questions and Answers

[0851] The user types in the chat box, "When is this assignment due?" The server receives this message, and the AI ​​moderator announces, "There is a new question. Sato-san, please answer."

[0852] Prompt Sentence Examples

[0853] "How should I address a participant who seems nervous during a remote meeting?"

[0854] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0855] Step 1:

[0856] Meeting planning

[0857] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda. This is done by using the Google Calendar API, Microsoft Exchange API, and also by reading data from an internal database. Specifically, it sends a request to the API endpoint and parses the data returned as a response to retrieve the meeting information. This information is the input data, and the retrieved date and time of the meeting, the participant list, and the agenda are the output data.

[0858] Step 2:

[0859] Send a reminder

[0860] The server generates reminders based on the scheduled date and time, participant list, and agenda of the meeting. These reminders are sent to participants via email or notification system via Gmail API or Slack API. The input data is the retrieved meeting information, and the output data is the reminder sending status.

[0861] Step 3:

[0862] Generate a Conference Link

[0863] When the meeting start time approaches, the server generates a new meeting link using the Zoom API, Microsoft Teams API, or Google Meet API. It sends an appropriate request to the API and obtains the meeting link as a response. The input data is the meeting start time, and the output data is the generated meeting link.

[0864] Step 4:

[0865] Meeting link notification

[0866] The server notifies participants of the generated meeting link via email or chat tool. The Slack API is used to notify all participants of the link. The input data is the generated meeting link, and the output data is the notification sending status.

[0867] Step 5:

[0868] AI moderator activation

[0869] The server starts an AI moderator program at the same time as the start of the meeting. The moderator program manages the progress of the meeting using a generative AI model (e.g., GPT-3). The input data is the start signal for the meeting, and the output data is the start status of the AI ​​moderator program.

[0870] Step 6:

[0871] Sending a request to speak

[0872] A user uses a terminal to send a speech request, and the terminal sends this request to the server. The input data is the user's speech request, and the output data is the request status sent to the server.

[0873] Step 7:

[0874] Receiving and organizing requests to speak

[0875] The server organizes speech requests received from terminals in order, adds the requests to a queue, and selects the next user to speak. The input data is the received speech requests, and the output data is the organized speech order.

[0876] Step 8:

[0877] Speaking order notification

[0878] The server notifies the AI ​​moderator, "The next speaker is Mr. / Ms. X." The input data is the organized order of speeches, and the output data is the notification sending status.

[0879] Step 9:

[0880] Meeting transcription

[0881] The server transcribes the meeting content in real time using a speech recognition engine such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The input data is the meeting audio, and the output data is the transcribed text.

[0882] Step 10:

[0883] Summarizing and sharing meeting content

[0884] The server processes the transcribed text with a summarization algorithm to extract key points, and shares the extracted summary with participants via email or document format. The input data is the transcribed text, and the output data is the summarized text and the sharing status.

[0885] Step 11:

[0886] Receiving and parsing questions

[0887] The user enters a question using the chat function. The server receives the question, and the AI ​​moderator analyzes the question and notifies the person responsible for answering it. The input data is the question entered in the chat, and the output data is the analysis result and notification status.

[0888] Step 12:

[0889] Action item confirmation and notifications

[0890] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. The input data is the meeting summary and task list, and the output data is the notification status.

[0891] Step 13:

[0892] Sending follow-up notifications

[0893] Even after the meeting ends, the server periodically sends follow-up notifications. It uses reminder emails and task management tools to ensure that action items are carried out. The input data is a list of action items, and the output data is the sending status of the follow-up notifications.

[0894] Step 14:

[0895] Emotion engine analysis and response generation

[0896] The server uses an emotion engine to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the results of this analysis, the AI ​​moderator generates appropriate responses and comments. For example, if the server detects that the user is nervous, it generates a message such as "Please relax and speak." The input data is the user's emotional state, and the output data is the response message.

[0897] (Application example 2)

[0898] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0899] With conventional online conference systems, planning, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting are all done manually, making it difficult to manage efficiently. Furthermore, the meeting cannot proceed taking into account the emotional state of the participants, resulting in insufficient communication. This results in issues such as meetings not proceeding smoothly and participants' opinions not being fully elicited.

[0900] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the scheduled date and time of the meeting, the participant list, and the agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating AI moderation, receiving speech requests, and organizing the speaking order, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into the chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, means for sending follow-up notifications after the end of the meeting, and means for collecting and analyzing emotional data and generating appropriate responses and comments according to the emotional states of the participants. This not only automates the entire process of online meetings, but also enables appropriate responses according to the emotional states of the participants, thereby achieving efficient and effective meeting progress.

[0901] The "scheduled date and time of the meeting" refers to the specific date and time when the meeting will start.

[0902] The "participant list" indicates a list of users who are scheduled to participate in the conference.

[0903] An "agenda" is a list of specific topics or issues to be discussed at a meeting.

[0904] "Means for sending reminders" refers to a method or system for notifying users of upcoming meetings.

[0905] "Conference link" refers to the URL and connection information required for a user to participate in a conference.

[0906] "AI moderation" refers to a system that uses artificial intelligence to manage the progress of online meetings and call on speakers at appropriate times.

[0907] "Means for receiving requests to speak and organizing the order in which they speak" refers to a method or system for receiving requests to speak from participants and managing them in an appropriate order.

[0908] "Means for transcribing meeting content in real time" refers to technology that instantly converts audio during a meeting into text.

[0909] "Means for extracting a summary of the meeting content and sharing it with participants" refers to a method or system for summarizing the important points and main points from the meeting and communicating them to participants.

[0910] "Means of entering questions into chat, analyzing the questions, and notifying participants" refers to a method of receiving text questions from participants, understanding their content, and notifying participants at an appropriate time.

[0911] An "action item" refers to a specific task or work decided during a meeting.

[0912] "Means for sending follow-up notifications" refers to a method or system for checking whether action items are being properly carried out after the meeting and sending reminders as necessary.

[0913] "Emotion data" refers to information about the user's emotional state that can be inferred from facial expressions, tone of voice, and the like.

[0914] "Means for generating appropriate responses and comments" refers to methods and systems for providing necessary support and encouragement to participants based on emotional data.

[0915] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and combines it with an emotion engine that recognizes the user's emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[0916] System programs and their processing

[0917] Hardware and Software

[0918] Hardware: Servers, devices (smartphones, PCs, etc.)

[0919] Software: SQLite (database), smtplib (Python library), email.mime (Python library), speech recognition engine (e.g., Google Speech-to-Text API), sentiment analysis engine (e.g., Amazon Comprehend)

[0920] Meeting planning

[0921] The server retrieves the scheduled meeting date and time, the participant list and the agenda from the database and sends reminders to participants, which are sent via email.

[0922] Specific examples

[0923] For a meeting scheduled for next Monday, the server retrieves the scheduled date and time, the attendee list, and the agenda from the database, and sends email reminders to the attendees, helping them remember to plan for the meeting.

[0924] Example prompt sentence:

[0925] "Get the date, time, participant list, and agenda for your next seminar and send reminder emails to each participant."

[0926] Starting a meeting

[0927] When the meeting starts, the server generates a meeting link and notifies the participants, who can then join the meeting using the generated URL.

[0928] Speech Management

[0929] Users send requests to speak through their devices, which are received by the server and sorted in the appropriate order. The AI ​​moderator announces the order of speaking and helps ensure the meeting proceeds smoothly.

[0930] Specific examples

[0931] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[0932] Record and share meeting content

[0933] The server transcribes what is said during the meeting in real time, and a summary is extracted using a summarization algorithm and shared with participants, allowing all participants to accurately understand the content of the meeting.

[0934] Managing Questions and Answers

[0935] During the meeting, users can enter questions into the chat through their devices. The questions are received by the server, and the AI ​​moderator notifies them at the appropriate time. This function allows questions and answers to be exchanged smoothly.

[0936] Specific examples

[0937] During a meeting, a user enters a question via chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Mr. Sato, please answer."

[0938] Example prompt sentence:

[0939] "Receive chat questions and notify us at the right time"

[0940] Review and follow up on action items

[0941] The server checks for action items and notifies participants before the end of the meeting, and also sends follow-up notifications after the meeting to ensure that action items are carried out.

[0942] Emotion Engine Functions

[0943] By incorporating an emotion engine, the server analyzes the user's emotional state in real time during the meeting, allowing the AI ​​moderator to reduce participants' stress and maintain a good meeting atmosphere.

[0944] Specific examples

[0945] During a meeting, the emotion engine analyzes the user's level of tension from their facial expressions and tone of voice, and the AI ​​moderator encourages them to "relax and speak."

[0946] As described above, by using the above-mentioned hardware and software, the online conference system of the present invention can automate the entire conference process and respond appropriately to the emotional state of the participants, thereby realizing efficient and effective conference proceedings.

[0947] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0948] Step 1:

[0949] The server retrieves the scheduled date and time, participant list, and agenda of the meeting from the database. It receives a meeting ID as input and searches the database for the corresponding scheduled date and time, participant list, and agenda of the meeting and outputs it. This data is used as information required for subsequent processing.

[0950] Step 2:

[0951] The server sends reminders to participants based on the information it has obtained. It receives the participant list, scheduled date and time, and agenda as input, creates and sends an email, and receives a confirmation that the reminder was sent as output. The reminder helps participants remember that the meeting is happening.

[0952] Step 3:

[0953] When the meeting start time arrives, the server generates a meeting link and notifies the participants. It receives the meeting ID as input and generates a meeting link (URL). The generated URL is sent as output to the participants via email. This prepares the participants to join the meeting.

[0954] Step 4:

[0955] The server starts the AI ​​moderator, receives speaking requests, and organizes the speaking order. It receives speaking requests as input and organizes the speaking order. It adds the requests to the queue in the appropriate order. It outputs organized speaking order information. This allows the meeting to proceed smoothly.

[0956] Step 5:

[0957] The server transcribes what is said during the meeting in real time. It receives the voice data from the meeting as input and converts it into text data using a speech recognition engine. It obtains the text data generated in real time as output. This records the contents of the meeting.

[0958] Step 6:

[0959] The server extracts a summary of the meeting content and shares it with participants. It receives the transcribed text data as input and uses a summarization algorithm to extract key points. It generates the extracted summary as output and shares it with participants, thereby conveying the important points of the meeting to participants.

[0960] Step 7:

[0961] Users input questions through chat during a meeting. The text-based question is received as input and sent to the server. The server analyzes the question and notifies the participants. The analyzed question is obtained as output, and the AI ​​moderator makes notifications based on that. This allows for a smooth exchange of questions and answers.

[0962] Step 8:

[0963] Before the end of the meeting, the server confirms the action items and notifies the participants. It receives as input the action items decided during the meeting and notifies the participants. It outputs a confirmation that the notification was sent, ensuring that the meeting decisions are carried out.

[0964] Step 9:

[0965] After the meeting ends, the server sends a follow-up notification. It receives as input the status of the post-meeting action item, creates and sends a reminder, and as output, provides confirmation that the follow-up notification was sent, ensuring that the action item is followed up appropriately.

[0966] Step 10:

[0967] The server collects and analyzes emotional data, generating appropriate responses and comments according to the participants' emotional state. It receives emotional data such as the user's facial expressions and tone of voice as input, and analyzes it using an emotion analysis engine. It generates appropriate responses and comments based on the analysis results as output, and notifies the participants. This reduces stress for participants and maintains a good meeting atmosphere.

[0968] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0969] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0970] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0971] [Third embodiment]

[0972] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0973] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0974] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0975] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0976] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0977] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0978] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0979] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0980] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0981] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0982] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0983] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0984] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting.

[0985] System programs and their processing

[0986] 1. Planning a meeting

[0987] First, the server retrieves the scheduled meeting date and time, the list of attendees, and the agenda. This information can be retrieved from a calendar API or a database. Based on this information, the server sends reminders to attendees. These reminders can be sent via email or a notification system.

[0988] 2. Starting the meeting

[0989] Next, when the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. When the meeting starts, the server launches an AI moderator to manage the progress of the meeting.

[0990] 3. Comment Management

[0991] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[0992] 4. Record and share content

[0993] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[0994] 5. Question and Answer Management

[0995] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[0996] 6. Review and follow up on action items

[0997] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[0998] Specific examples

[0999] Example 1: Planning a meeting

[1000] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday. With this information, the server sends reminders to participants via email, ensuring they don't forget the meeting date.

[1001] Example 2: Speech Management

[1002] During a meeting, a user sends a request to speak from their device. The device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will speak next."

[1003] Example 3: Managing questions and answers

[1004] During a meeting, a user can enter a question via chat. The server receives the question, and the AI ​​moderator responds by saying, "A new question has arrived. Mr. Sato, please answer." This function ensures a smooth flow of questions and answers.

[1005] In this way, the system of the present invention can automate the entire process of an online conference, enabling the conference to proceed efficiently and effectively.

[1006] The processing flow will be explained below.

[1007] Step 1:

[1008] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, either from a calendar API or a database.

[1009] Step 2:

[1010] The server uses the information it retrieves to send reminders to participants, either via email or a notification system.

[1011] Step 3:

[1012] When the meeting starts, the server generates a meeting link, which is a specific URL.

[1013] Step 4:

[1014] The server notifies participants of the generated conference link via email or chat.

[1015] Step 5:

[1016] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[1017] Step 6:

[1018] During a conference, users can send requests to speak through their devices, using buttons or the chat function.

[1019] Step 7:

[1020] The device sends a request to the server, which sends a POST request to the server's API.

[1021] Step 8:

[1022] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[1023] Step 9:

[1024] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[1025] Step 10:

[1026] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[1027] Step 11:

[1028] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[1029] Step 12:

[1030] The server shares the summary with participants via email or cloud storage.

[1031] Step 13:

[1032] During a meeting, a user enters a question in the chat box and presses the send button.

[1033] Step 14:

[1034] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[1035] Step 15:

[1036] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[1037] Step 16:

[1038] The server notifies participants of action items, either via email or through a notification system.

[1039] Step 17:

[1040] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[1041] Example 1

[1042] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1043] Conventional online conference systems require manual management and progress of meetings, which requires a lot of time and effort. Furthermore, there are often ineffective management of the order of comments, recording of discussions, and management of questions and answers during meetings, which reduces the efficiency of meetings. Furthermore, insufficient follow-up after the meeting can lead to delays in the implementation of action items.

[1044] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1045] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating an AI moderator, receiving speech requests, and organizing the speaking order, means for receiving speech requests from terminals and adding them to a queue, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into a chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, and means for sending follow-up notifications after the end of the meeting. This automates the entire process of online meetings, enabling efficient and effective meeting conduct.

[1046] "Scheduled date and time of the meeting" refers to the specific date and time for which the meeting is scheduled.

[1047] "Participant List" means a list of all participants invited to a Conference.

[1048] An "agenda" is a list of items or topics to be discussed at a meeting.

[1049] "Means for sending reminders" refers to a function that notifies meeting participants so they don't forget about the meeting.

[1050] "Means for generating a meeting link" refers to the function for creating a URL for joining an online meeting.

[1051] "Means for activating AI moderation" refers to the function of activating artificial intelligence that automatically manages the progress of the meeting.

[1052] "Means for receiving a speech request" refers to a function for receiving a request to speak from a participant.

[1053] "Means for organizing the speech order" refers to a function for arranging received speech requests in an appropriate order.

[1054] "Means to add to queue" refers to the ability to add a request to speak to a waiting list.

[1055] "Means of real-time transcription" refers to the ability to record audio during a meeting as text in real time.

[1056] "Means of extracting a summary" refers to the ability to extract key points and important information from the entire content of a meeting.

[1057] "Means for entering questions into chat" refers to the function by which participants enter questions using the chat function.

[1058] "Means for analyzing questions and notifying participants" refers to the function of analyzing the entered questions and notifying the relevant participants at the appropriate time.

[1059] "Means for checking action items" refers to the function for checking specific action items decided in the meeting.

[1060] "Means for sending follow-up notifications" refers to a function for sending follow-up notifications to participants after the end of a conference.

[1061] This invention relates to a system that uses AI to appropriately moderate online meetings and promote communication. This system automates a series of processes, from planning the meeting to proceeding with the meeting, recording and sharing the content, managing questions and answers, and following up after the meeting. Specific embodiments of this system are described below.

[1062] 1. Planning a meeting

[1063] 1.1 Getting the scheduled date, time, attendee list, and agenda

[1064] The server uses a calendar API or database to retrieve the scheduled date and time of the meeting, the attendee list, and the agenda. Specifically, it could use the Google Calendar API or Microsoft Outlook API. This method allows accurate acquisition of basic meeting information.

[1065] 1.2 Sending reminders

[1066] The server sends reminders to participants based on the information obtained, typically using an email sending API (e.g., SendGrid) or push notifications.

[1067] 2. Starting the meeting

[1068] 2.1 Generating and notifying the meeting link

[1069] When the meeting starts, the server generates a meeting link in a specific URL format and notifies the participants. The meeting link is generated using the API of a common online meeting platform (e.g., Zoom Meetings API).

[1070] 2.2 Starting AI moderation

[1071] When the meeting starts, the server launches an AI moderator to manage the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the meeting.

[1072] 3. Comment Management

[1073] 3.1 Sending a request to speak

[1074] Users send requests to speak using the buttons on their devices or the chat function, and this information is sent from the device to the server.

[1075] 3.2 Receiving requests and organizing the order of speaking

[1076] The server receives the requests and organizes the speaking order. The request is added to the queue, and the AI ​​moderator announces, "The next speaker is Mr. / Ms. XX."

[1077] 4. Record and share content

[1078] 4.1 Real-time transcription

[1079] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[1080] 4.2 Abstract Extraction and Notification

[1081] The server extracts a summary of the meeting contents and shares it with participants. It uses a summary generation model (e.g., BART) to extract the main points of the meeting and notifies them via email or other means.

[1082] 5. Question and Answer Management

[1083] 5.1 Entering a Question

[1084] Users can input questions using the chat function on their devices, and this information is sent from the devices to the server.

[1085] 5.2 Question notification and response promotion

[1086] The server receives the question, and the AI ​​moderator prompts the answer at the appropriate time, notifying the user, "There is a new question. Mr. / Ms. XX, please answer."

[1087] 6. Review and follow up on action items

[1088] 6.1 Extracting Action Items

[1089] Before the end of the meeting, the server checks the action items and extracts any uncompleted tasks or action targets.

[1090] 6.2 Follow-up Notifications

[1091] The server will send periodic follow-up notifications after the meeting to help ensure action items are carried out.

[1092] Specific examples

[1093] Example 1: Send a reminder

[1094] The server retrieves meeting information scheduled for "October 15, 2023 at 2:00 PM" from the Google Calendar API and sends reminders to participants using the SendGrid API.

[1095] Example 2: Speech Management

[1096] During a meeting, when a user clicks a button on their device to send a request to speak, the server receives the request and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[1097] Example 3: Managing questions and answers

[1098] During a meeting, if a user types in chat, "More details about this topic?", the server receives the question and the AI ​​moderator notifies them, "There is a new question. Mr. Sato, please answer."

[1099] Prompt Sentence Examples

[1100] Please tell me more about how to notify "X-Person will be speaking next."

[1101] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1102] Step 1: Get the scheduled meeting date, time, attendee list, and agenda

[1103] The server retrieves the scheduled date and time of the meeting, the attendee list, and the agenda from a calendar API (e.g., Google Calendar API) or a database. In this step, the server confirms the meeting date, attendees, and agenda based on the data retrieved through the API. The input is information from the calendar API or database, and the output is the meeting information.

[1104] Specific behavior:

[1105] The server sends a request to the Google Calendar API to retrieve the meeting information.

[1106] The server reads the participant list from a database.

[1107] Step 2: Send a reminder

[1108] The server sends reminders based on the acquired meeting information. This process uses an email sending API (e.g., SendGrid) or a push notification service to notify participants. The input is the meeting information acquired in step 1, and the output is the sent reminder notification.

[1109] Specific behavior:

[1110] The server uses the SendGrid API to send a reminder to participants saying, "We're planning a meeting for October 15th."

[1111] The server uses a push notification service to send notifications to participants' devices.

[1112] Step 3: Generate and share the meeting link

[1113] When the meeting start time arrives, the server uses the API of the online meeting platform (e.g., Zoom Meetings API) to generate a meeting link and notify participants. The input is the meeting start time and participant information, and the output is the generated meeting link and its notification.

[1114] Specific behavior:

[1115] The server uses the Zoom Meetings API to generate the meeting link.

[1116] The server will send the generated link to participants via email and push notification.

[1117] Step 4: Start the AI ​​moderator

[1118] The server starts the AI ​​moderator as soon as the meeting starts and manages the progress of the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the progress of the meeting. The input is the trigger to start the meeting, and the output is the startup status of the AI ​​moderator.

[1119] Specific behavior:

[1120] The server launches OpenAI GPT-3 at the start of the meeting and begins moderating.

[1121] Step 5: Sending and receiving requests to speak

[1122] Users send speech requests using the buttons or chat function on their devices, and the devices send this information to the server. The server receives the requests and organizes the speech order. The input is the speech request from the user, and the output is the organized speech order.

[1123] Specific behavior:

[1124] The user clicks the "Request to speak" button on the device.

[1125] The terminal sends a request to speak to the server.

[1126] The server receives the request and adds it to a queue.

[1127] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. XX."

[1128] Step 6: Real-time transcription

[1129] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is the audio data during the meeting, and the output is the transcribed text.

[1130] Specific behavior:

[1131] The server sends the audio data to the Google Cloud Speech-to-Text API, which returns it as text.

[1132] Transcribe what is being said in real time.

[1133] Step 7: Extract and notify the summary

[1134] The server extracts a summary of the meeting content and notifies participants. The summary is created using a summary generation model (e.g., BART). The input is the transcribed text, and the output is the summarized text.

[1135] Specific behavior:

[1136] The server uses the BART model to extract summaries from the transcribed content.

[1137] The server will notify participants of the summary via email.

[1138] Step 8: Manage questions and answers

[1139] Users input questions into the chat through their devices, the server receives and analyzes the questions, and the AI ​​moderator facilitates the answers. The input is the user's question, and the output is the analyzed question and answer instructions.

[1140] Specific behavior:

[1141] The user types a question into the chat box.

[1142] The terminal sends a query to the server.

[1143] The server notifies the AI ​​moderator of the question and prompts them, "There is a new question. Mr. / Ms. XX, please answer."

[1144] Step 9: Review and follow up on action items

[1145] The server checks action items before the end of the meeting, extracts uncompleted tasks and action targets, and notifies participants. It also periodically sends follow-up notifications after the meeting ends. The input is the action items recorded during the meeting, and the output is confirmation and follow-up notifications.

[1146] Specific behavior:

[1147] The server checks the task list for the meeting and extracts any uncompleted items.

[1148] The server will send follow-up notifications one week after the meeting ends.

[1149] (Application example 1)

[1150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1151] Currently, maintaining efficiency and consistency in online meetings and virtual events is difficult, particularly in managing participants' opinions and smoothly managing the flow of real-time questions and answers. Furthermore, in environments with a large number of participants, such as online product briefings and campaign events held in virtual stores, managing speaking requests and questions becomes even more cumbersome, hindering effective communication.

[1152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1153] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, and means for managing speech requests and questions. This enables efficient and consistent management of the flow of speech requests and questions at online product briefings and campaign events in virtual stores, enabling effective communication.

[1154] The "scheduled date and time of the meeting" refers to the date and time when the meeting will start.

[1155] A "participant list" refers to a list of people who plan to attend a meeting or event.

[1156] An "agenda" is a list of topics or items to be discussed at a meeting or event.

[1157] "Means for sending reminders" means a function or system for sending advance notice of a meeting or event to participants.

[1158] "Conference Link" means a specific web address or invitation link through which participants can join a conference or event.

[1159] "Means of notification" refers to a communication means for conveying information such as a conference link to participants.

[1160] "AI moderation" refers to a system that uses artificial intelligence to automatically manage the progress of meetings and events.

[1161] "Request to speak" means a request by a participant for permission to speak during a meeting or event.

[1162] "Means for organizing the order of speech" refers to a system or function that accepts speech requests from participants and arranges the speech in the appropriate order.

[1163] "Means of real-time transcription" refers to functions or systems that convert statements made during meetings or events into text data in real time.

[1164] "Means to extract summaries and share them with participants" refers to a function for summarizing the contents of a meeting or event and conveying the main points to participants.

[1165] "Means for analyzing questions and notifying participants" refers to a system or function for analyzing questions entered in chat and informing the appropriate participants of their content.

[1166] "Means to review action items and notify participants" refers to the ability to review outstanding tasks and action items before the end of a meeting or event and notify participants as needed.

[1167] "Means for sending follow-up notifications" refers to a system or function for checking the progress of action items after a meeting or event and sending follow-up notifications to participants.

[1168] "Virtual store" refers to a virtual store that operates on the Internet, an online sales and service platform that does not have an actual physical location.

[1169] An "online product briefing" refers to a product briefing event held over the Internet.

[1170] "Campaign Event" means a series of online events conducted to promote a particular product or service.

[1171] "Means for managing speech requests and questions" refers to a system or function for managing speech requests and questions from participants during an event in a virtual store.

[1172] This invention relates to a system and method for smoothly conducting online product briefings and campaign events in a virtual store. This system uses AI to automate the event moderation and provides functions for managing participants' comments and smoothly managing the flow of questions and answers. The implementation of this system includes the following specific configuration and operation procedures.

[1173] Hardware and software used

[1174] server:

[1175] Role: Overall moderator, sending reminders, generating meeting links, managing speaking requests and questions, real-time transcription, extracting summaries, and following up notifications.

[1176] Examples: Google Cloud Platform, AWS (Amazon Web Services)

[1177] Device:

[1178] Role: A device for participants to send requests to speak and type questions through chat.

[1179] Examples: smartphones, tablets, and PCs

[1180] Server processing

[1181] 1. Send reminders:

[1182] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda, and sends reminders to participants, allowing them to be aware of the meeting or event in advance.

[1183] Example: A server sends reminder emails to participants on the weekend about next week's event.

[1184] 2. Conference link generation and notification:

[1185] At the start time of the meeting, the server generates a meeting link and notifies participants, who can use this link to join the online event.

[1186] Example: Before the event starts, a meeting link is generated and sent to participants' email addresses.

[1187] 3. Managing requests and questions:

[1188] Participants can send requests to speak through their devices, and the server adds them to a queue and organizes them in the appropriate order.

[1189] Additionally, when a question is sent using the chat function, the server analyzes it and notifies the appropriate participant.

[1190] Example: During an event, a participant sends a request to speak, and the server organizes the order and notifies them, "Person A will speak next." Also, a question is sent via chat, and the server notifies them, "There is a new question. Person B, please answer."

[1191] 4. Real-time transcription and summary extraction:

[1192] The server transcribes speeches made during the event in real time, and a summarization algorithm is used to extract key points and share them with participants.

[1193] Example: After the event, a transcript and summary of what was said is automatically emailed to attendees.

[1194] 5. Follow-up Notification:

[1195] Before the event ends, the server checks for action items, extracts any incomplete tasks and action items, and notifies participants. After the event ends, the server periodically sends follow-up notifications to ensure that action items are carried out.

[1196] Example: After the event ends, the server notifies participants that "Uncompleted task: Create explanatory materials about the product details."

[1197] Examples of prompt statements

[1198] Implement a function send_reminder to send reminders to participants. It uses Gmail's SMTP server and takes the specified email address, subject, and body as arguments. It also requires user authentication to send the email.

[1199] Through the above process, the present invention can realize effective communication and smooth event progress within the virtual store.

[1200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1201] Step 1:

[1202] The server obtains the scheduled date and time of the meeting, the participant list, and the agenda. This information is obtained as input from a database or calendar API. The output is the obtained scheduled date and time, participant list, and agenda data. Specifically, the server calls the API to extract the data and saves it in an internal database.

[1203] Step 2:

[1204] The server sends reminders to participants based on the information it obtains. The input is the scheduled date and time of the meeting, the participant list, and the agenda. The output is a reminder email or notification sent to participants. Specifically, the server generates the reminder content and sends the email using an SMTP server.

[1205] Step 3:

[1206] At the start time of the conference, the server generates a conference link and notifies the participants. The input is the start time and the participant list. The output is the generated conference link and an email notifying the participants. Specifically, the server generates a conference URL and sends a notification via email.

[1207] Step 4:

[1208] A user uses a terminal to send a speech request. The input is the user's request information. The output is adding the request information to the speech queue by sending it to the server. In concrete terms, the user inputs a request through the interface of the terminal, and the terminal sends the information to the server.

[1209] Step 5:

[1210] The server receives speech requests and organizes the speech order. The input is request information from the user. The output is an organized speech order list. Specifically, the server adds requests to a queue and manages the requests in the appropriate order.

[1211] Step 6:

[1212] A user inputs a question into the chat, and the terminal sends the question to the server. The input is the question entered by the user. The output is the question being sent to the server. In concrete terms, the user inputs a question into the chat interface and presses the send button.

[1213] Step 7:

[1214] The server analyzes the question and notifies the appropriate participant. The input is the question sent by the user via chat. The output is data used to analyze and notify the question. Specifically, the server uses natural language processing to analyze the question and notify the appropriate respondent.

[1215] Step 8:

[1216] The server transcribes the meeting content in real time and extracts a summary. The input is the audio data of the meeting. The output is the transcribed text data and summary data. Specifically, the server uses a speech recognition engine to convert the audio data into text and then summarizes the text using a summarization algorithm.

[1217] Step 9:

[1218] The server checks action items and notifies participants before the end of the meeting. The input is the list of action items recorded during the meeting. The output is to notify participants of incomplete action items. Specifically, the server scans the action item list and adds incomplete tasks to the notification list.

[1219] Step 10:

[1220] The server sends follow-up notifications after the meeting ends. The input is the progress of the action items at the time the meeting ends. The output is a follow-up notification email or notification. Specifically, the server checks the status of the action items and periodically sends follow-up notifications.

[1221] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1222] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and further combines it with an emotion engine that recognizes user emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[1223] System programs and their processing

[1224] 1. Planning a meeting

[1225] First, the server retrieves the scheduled meeting date and time, the attendee list, and the agenda. This information can be retrieved from a calendar API or a database. Then, based on this information, the server sends reminders to the attendees. Reminders can be sent via email or a notification system.

[1226] 2. Starting the meeting

[1227] When the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. The server then launches an AI moderator to manage the meeting.

[1228] 3. Comment Management

[1229] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[1230] 4. Record and share content

[1231] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[1232] 5. Question and Answer Management

[1233] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[1234] 6. Review and follow up on action items

[1235] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[1236] 7. Emotion Engine Functions

[1237] By incorporating an emotion engine, the server can analyze the user's emotional state in real time during a meeting. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, and speech content, and analyzes and stores these as data.

[1238] The AI ​​moderator can generate appropriate responses and comments based on this emotional data, reducing stress for participants and maintaining a good meeting atmosphere. For example, if a user is nervous, the AI ​​moderator will say, "Please relax and speak."

[1239] Specific examples

[1240] Example 1: Planning a meeting and sending reminders

[1241] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday and sends reminders to participants via email, reminding them to remember the meeting date.

[1242] Example 2: Speech management and emotion recognition

[1243] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to the queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next." At the same time, the emotion engine analyzes the user's emotional state, and if they appear tense, it will say, "Please relax and speak."

[1244] Example 3: Question and answer management and sentiment analysis

[1245] During a meeting, a user types a question into the chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Sato-san, please answer." At the same time, the emotion engine analyzes Sato-san's anxious expression and provides additional support, such as, "If the question is complicated, please speak more slowly."

[1246] In this way, the system of the present invention not only automates the entire process of online meetings, enabling efficient and effective meeting conduct, but also supports more human and interactive meetings through the introduction of an emotion engine.

[1247] The processing flow will be explained below.

[1248] Step 1:

[1249] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, which can be retrieved from a calendar API or a database.

[1250] Step 2:

[1251] The server uses the information it retrieves to send reminders to participants, which can be sent via email or a notification system.

[1252] Step 3:

[1253] When the meeting starts, the server generates a meeting link, which is a specific URL.

[1254] Step 4:

[1255] The server notifies participants of the generated conference link via email or chat.

[1256] Step 5:

[1257] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[1258] Step 6:

[1259] Users send requests to speak through their devices, using buttons or the chat function.

[1260] Step 7:

[1261] The device sends a request to the server, which sends a POST request to the server's API.

[1262] Step 8:

[1263] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[1264] Step 9:

[1265] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[1266] Step 10:

[1267] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[1268] Step 11:

[1269] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[1270] Step 12:

[1271] The server shares the summary with participants via email or cloud storage.

[1272] Step 13:

[1273] During a meeting, a user enters a question in the chat box and presses the send button.

[1274] Step 14:

[1275] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[1276] Step 15:

[1277] During the meeting, the server uses an emotion engine to analyze the user's emotional state by detecting facial expressions, tone of voice, and speech content and analyzing the data.

[1278] Step 16:

[1279] The emotion engine analyzes the user's emotional state and sends the results back to the server. The emotional data is then passed on to the AI ​​host.

[1280] Step 17:

[1281] Based on the emotional data, the AI ​​moderator generates appropriate comments, such as "Please relax and speak," and notifies the speaker in real time through voice synthesis and text display.

[1282] Step 18:

[1283] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[1284] Step 19:

[1285] The server notifies participants of action items, either via email or through a notification system.

[1286] Step 20:

[1287] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[1288] Example 2

[1289] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1290] Ensuring smooth online meetings and smooth communication between participants requires a lot of manual work. This includes planning meetings, sending reminders, managing comments, recording and sharing content, coordinating questions and answers, and following up after the meeting. It is also important to understand participants' emotional states and communicate appropriately. However, performing these tasks manually is labor-intensive and inefficient. Furthermore, analyzing participants' emotions in real time and responding appropriately is practically difficult. Therefore, there is a need for an automated system to solve these issues and improve the efficiency and quality of online meetings.

[1291] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1292] In this invention, the server includes: means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda; means for sending reminders to participants; means for generating a meeting link at the start time of the meeting and notifying participants; means for activating an AI moderator, receiving speaking requests, and organizing the speaking order; means for transcribing the meeting content in real time; means for extracting a summary of the meeting content and sharing it with participants; means for inputting questions into a chat, analyzing the questions, and notifying participants; means for checking action items before the end of the meeting and notifying participants; means for sending follow-up notifications after the end of the meeting; and means for analyzing user emotions in real time using an emotion engine and generating appropriate responses and comments. This automates the entire process of online meetings, enabling efficient and effective meeting conduct and also realizing smoother communication by understanding participants' emotions in real time and responding appropriately.

[1293] The "scheduled date and time of the conference" refers to the specific date and time when the conference is scheduled to start.

[1294] A "participant list" is a list that includes information such as names and contact details of people who will be attending a conference.

[1295] An agenda is a list of the main topics and subjects of a meeting that is compiled in advance and used as a guideline for proceeding.

[1296] A "reminder" is an email or message that notifies participants in advance so they don't forget about the meeting schedule.

[1297] "Meeting Link" means a specific URL used by an online meeting platform that allows participants to join a meeting by clicking on the link.

[1298] "AI moderation" is a system that uses artificial intelligence to manage the progress of a meeting, automatically coordinating remarks and managing questions.

[1299] A "request to speak" is a request made by a participant who wishes to speak during a conference to inform the moderator of that request.

[1300] "Speaking order" refers to the order in which people speak in a meeting, and is adjusted to ensure smooth progress.

[1301] "Transcription" is the process or result of converting audio from a meeting into text in real time.

[1302] A "summary" is a concise summary of the main points and conclusions discussed at a meeting that is shared with all participants.

[1303] "Chat" is a real-time means of communication using text messages.

[1304] "Action items" refer to specific actions or tasks decided during a meeting.

[1305] "Follow-up notifications" are reminders sent to participants after the meeting to ensure action items are carried out.

[1306] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, speech content, etc. to recognize their emotional state and generate an appropriate response.

[1307] "Appropriate responses and comments" are statements and feedback generated by AI based on the analyzed emotional state, designed to reduce participants' stress and create a positive atmosphere in the meeting.

[1308] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. It also incorporates an emotion engine that recognizes the user's emotions. This system automates a series of processes, from planning the meeting, conducting the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[1309] First, the server retrieves the scheduled meeting date and time, the list of participants, and the agenda. This information can be retrieved from a calendar API, such as the Google Calendar API or the Microsoft Exchange API, or from an internal database. Based on this information, the server generates and sends reminders to participants via email or a notification system (for example, Slack or Microsoft Teams).

[1310] Next, when the meeting's start time approaches, the server generates a meeting link to be used in an online meeting platform (such as Zoom, Microsoft Teams, or Google Meet) and notifies participants via email or chat. At the same time, the server launches an AI moderator program to manage the meeting.

[1311] During a meeting, users use their devices (PC, tablet, smartphone) to send requests to speak. This request is sent using the device's buttons or chat function, and the device then sends it to the server. The server receives the requests and organizes the speaking order. The AI ​​moderator notifies participants, such as "Person XX will be speaking next," helping to ensure smooth progress.

[1312] The server transcribes what is said during the meeting in real time using engines such as Google Cloud Speech-to-Text, IBM Watson Speech to Text, and Microsoft Azure Cognitive Services for speech recognition. The server then summarizes the meeting, extracts key points, and shares them with participants via email or in a document format.

[1313] During the meeting, users can use the chat function to enter questions. The questions entered in the chat are received by the server, and the AI ​​moderator notifies them by saying, "There is a new question. Mr. / Ms. X, please answer."

[1314] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. After the meeting ends, the server periodically sends follow-up notifications to help ensure that action items are carried out. This follow-up is achieved by using reminder emails and integration with task management tools (Trello and Asana).

[1315] The system also incorporates an emotion engine. The server uses an emotion engine (such as Microsoft Azure Emotion API, Amazon Rekognition, or Google Cloud AI) to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the analysis results, the AI ​​moderator generates appropriate responses and comments to reduce user stress and maintain a good meeting atmosphere. For example, it might say to a nervous user, "Please relax and speak."

[1316] Specific examples

[1317] Meeting planning and reminders

[1318] The server calls the Google Calendar API to retrieve meeting information for the next week, and automatically sends reminders to all participants via Gmail based on the meeting date, time, and participant list.

[1319] Meeting link generation and notification

[1320] 15 minutes before the meeting starts, the server creates a new meeting using the Zoom API and generates a meeting link, which is then sent to participants via Slack.

[1321] Organizing requests to speak

[1322] The user clicks the "Request to speak" button on their PC. The device sends this request to the server, which adds it to the queue. The AI ​​moderator announces, "Tanaka will be speaking next." At the same time, the emotion engine analyzes Tanaka's facial expressions and, if she appears nervous, says, "Please relax and speak."

[1323] Managing Questions and Answers

[1324] The user types in the chat box, "When is this assignment due?" The server receives this message, and the AI ​​moderator announces, "There is a new question. Sato-san, please answer."

[1325] Prompt Sentence Examples

[1326] "How should I address a participant who seems nervous during a remote meeting?"

[1327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1328] Step 1:

[1329] Meeting planning

[1330] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda. This is done by using the Google Calendar API, Microsoft Exchange API, and also by reading data from an internal database. Specifically, it sends a request to the API endpoint and parses the data returned as a response to retrieve the meeting information. This information is the input data, and the retrieved date and time of the meeting, the participant list, and the agenda are the output data.

[1331] Step 2:

[1332] Send a reminder

[1333] The server generates reminders based on the scheduled date and time, participant list, and agenda of the meeting. These reminders are sent to participants via email or notification system via Gmail API or Slack API. The input data is the retrieved meeting information, and the output data is the reminder sending status.

[1334] Step 3:

[1335] Generate a Conference Link

[1336] When the meeting start time approaches, the server generates a new meeting link using the Zoom API, Microsoft Teams API, or Google Meet API. It sends an appropriate request to the API and obtains the meeting link as a response. The input data is the meeting start time, and the output data is the generated meeting link.

[1337] Step 4:

[1338] Meeting link notification

[1339] The server notifies participants of the generated meeting link via email or chat tool. The Slack API is used to notify all participants of the link. The input data is the generated meeting link, and the output data is the notification sending status.

[1340] Step 5:

[1341] AI moderator activation

[1342] The server starts an AI moderator program at the same time as the start of the meeting. The moderator program manages the progress of the meeting using a generative AI model (e.g., GPT-3). The input data is the start signal for the meeting, and the output data is the start status of the AI ​​moderator program.

[1343] Step 6:

[1344] Sending a request to speak

[1345] A user uses a terminal to send a speech request, and the terminal sends this request to the server. The input data is the user's speech request, and the output data is the request status sent to the server.

[1346] Step 7:

[1347] Receiving and organizing requests to speak

[1348] The server organizes speech requests received from terminals in order, adds the requests to a queue, and selects the next user to speak. The input data is the received speech requests, and the output data is the organized speech order.

[1349] Step 8:

[1350] Speaking order notification

[1351] The server notifies the AI ​​moderator, "The next speaker is Mr. / Ms. X." The input data is the organized order of speeches, and the output data is the notification sending status.

[1352] Step 9:

[1353] Meeting transcription

[1354] The server transcribes the meeting content in real time using a speech recognition engine such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The input data is the meeting audio, and the output data is the transcribed text.

[1355] Step 10:

[1356] Summarizing and sharing meeting content

[1357] The server processes the transcribed text with a summarization algorithm to extract key points, and shares the extracted summary with participants via email or document format. The input data is the transcribed text, and the output data is the summarized text and the sharing status.

[1358] Step 11:

[1359] Receiving and parsing questions

[1360] The user enters a question using the chat function. The server receives the question, and the AI ​​moderator analyzes the question and notifies the person responsible for answering it. The input data is the question entered in the chat, and the output data is the analysis result and notification status.

[1361] Step 12:

[1362] Action item confirmation and notifications

[1363] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. The input data is the meeting summary and task list, and the output data is the notification status.

[1364] Step 13:

[1365] Sending follow-up notifications

[1366] Even after the meeting ends, the server periodically sends follow-up notifications. It uses reminder emails and task management tools to ensure that action items are carried out. The input data is a list of action items, and the output data is the sending status of the follow-up notifications.

[1367] Step 14:

[1368] Emotion engine analysis and response generation

[1369] The server uses an emotion engine to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the results of this analysis, the AI ​​moderator generates appropriate responses and comments. For example, if the server detects that the user is nervous, it generates a message such as "Please relax and speak." The input data is the user's emotional state, and the output data is the response message.

[1370] (Application example 2)

[1371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1372] With conventional online conference systems, planning, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting are all done manually, making it difficult to manage efficiently. Furthermore, the meeting cannot proceed taking into account the emotional state of the participants, resulting in insufficient communication. This results in issues such as meetings not proceeding smoothly and participants' opinions not being fully elicited.

[1373] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the scheduled date and time of the meeting, the participant list, and the agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating AI moderation, receiving speech requests, and organizing the speaking order, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into the chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, means for sending follow-up notifications after the end of the meeting, and means for collecting and analyzing emotional data and generating appropriate responses and comments according to the emotional states of the participants. This not only automates the entire process of online meetings, but also enables appropriate responses according to the emotional states of the participants, thereby achieving efficient and effective meeting progress.

[1374] The "scheduled date and time of the meeting" refers to the specific date and time when the meeting will start.

[1375] The "participant list" indicates a list of users who are scheduled to participate in the conference.

[1376] An "agenda" is a list of specific topics or issues to be discussed at a meeting.

[1377] "Means for sending reminders" refers to a method or system for notifying users of upcoming meetings.

[1378] "Conference link" refers to the URL and connection information required for a user to participate in a conference.

[1379] "AI moderation" refers to a system that uses artificial intelligence to manage the progress of online meetings and call on speakers at appropriate times.

[1380] "Means for receiving requests to speak and organizing the order in which they speak" refers to a method or system for receiving requests to speak from participants and managing them in an appropriate order.

[1381] "Means for transcribing meeting content in real time" refers to technology that instantly converts audio during a meeting into text.

[1382] "Means for extracting a summary of the meeting content and sharing it with participants" refers to a method or system for summarizing the important points and main points from the meeting and communicating them to participants.

[1383] "Means of entering questions into chat, analyzing the questions, and notifying participants" refers to a method of receiving text questions from participants, understanding their content, and notifying participants at an appropriate time.

[1384] An "action item" refers to a specific task or work decided during a meeting.

[1385] "Means for sending follow-up notifications" refers to a method or system for checking whether action items are being properly carried out after the meeting and sending reminders as necessary.

[1386] "Emotion data" refers to information about the user's emotional state that can be inferred from facial expressions, tone of voice, and the like.

[1387] "Means for generating appropriate responses and comments" refers to methods and systems for providing necessary support and encouragement to participants based on emotional data.

[1388] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and combines it with an emotion engine that recognizes the user's emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[1389] System programs and their processing

[1390] Hardware and Software

[1391] Hardware: Servers, devices (smartphones, PCs, etc.)

[1392] Software: SQLite (database), smtplib (Python library), email.mime (Python library), speech recognition engine (e.g., Google Speech-to-Text API), sentiment analysis engine (e.g., Amazon Comprehend)

[1393] Meeting planning

[1394] The server retrieves the scheduled meeting date and time, the participant list and the agenda from the database and sends reminders to participants, which are sent via email.

[1395] Specific examples

[1396] For a meeting scheduled for next Monday, the server retrieves the scheduled date and time, the attendee list, and the agenda from the database, and sends email reminders to the attendees, helping them remember to plan for the meeting.

[1397] Example prompt sentence:

[1398] "Get the date, time, participant list, and agenda for your next seminar and send reminder emails to each participant."

[1399] Starting a meeting

[1400] When the meeting starts, the server generates a meeting link and notifies the participants, who can then join the meeting using the generated URL.

[1401] Speech Management

[1402] Users send requests to speak through their devices, which are received by the server and sorted in the appropriate order. The AI ​​moderator announces the order of speaking and helps ensure the meeting proceeds smoothly.

[1403] Specific examples

[1404] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[1405] Record and share meeting content

[1406] The server transcribes what is said during the meeting in real time, and a summary is extracted using a summarization algorithm and shared with participants, allowing all participants to accurately understand the content of the meeting.

[1407] Managing Questions and Answers

[1408] During the meeting, users can enter questions into the chat through their devices. The questions are received by the server, and the AI ​​moderator notifies them at the appropriate time. This function allows questions and answers to be exchanged smoothly.

[1409] Specific examples

[1410] During a meeting, a user enters a question via chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Mr. Sato, please answer."

[1411] Example prompt sentence:

[1412] "Receive chat questions and notify us at the right time"

[1413] Review and follow up on action items

[1414] The server checks for action items and notifies participants before the end of the meeting, and also sends follow-up notifications after the meeting to ensure that action items are carried out.

[1415] Emotion Engine Functions

[1416] By incorporating an emotion engine, the server analyzes the user's emotional state in real time during the meeting, allowing the AI ​​moderator to reduce participants' stress and maintain a good meeting atmosphere.

[1417] Specific examples

[1418] During a meeting, the emotion engine analyzes the user's level of tension from their facial expressions and tone of voice, and the AI ​​moderator encourages them to "relax and speak."

[1419] As described above, by using the above-mentioned hardware and software, the online conference system of the present invention can automate the entire conference process and respond appropriately to the emotional state of the participants, thereby realizing efficient and effective conference proceedings.

[1420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1421] Step 1:

[1422] The server retrieves the scheduled date and time, participant list, and agenda of the meeting from the database. It receives a meeting ID as input and searches the database for the corresponding scheduled date and time, participant list, and agenda of the meeting and outputs it. This data is used as information required for subsequent processing.

[1423] Step 2:

[1424] The server sends reminders to participants based on the information it has obtained. It receives the participant list, scheduled date and time, and agenda as input, creates and sends an email, and receives a confirmation that the reminder was sent as output. The reminder helps participants remember that the meeting is happening.

[1425] Step 3:

[1426] When the meeting start time arrives, the server generates a meeting link and notifies the participants. It receives the meeting ID as input and generates a meeting link (URL). The generated URL is sent as output to the participants via email. This prepares the participants to join the meeting.

[1427] Step 4:

[1428] The server starts the AI ​​moderator, receives speaking requests, and organizes the speaking order. It receives speaking requests as input and organizes the speaking order. It adds the requests to the queue in the appropriate order. It outputs organized speaking order information. This allows the meeting to proceed smoothly.

[1429] Step 5:

[1430] The server transcribes what is said during the meeting in real time. It receives the voice data from the meeting as input and converts it into text data using a speech recognition engine. It obtains the text data generated in real time as output. This records the contents of the meeting.

[1431] Step 6:

[1432] The server extracts a summary of the meeting content and shares it with participants. It receives the transcribed text data as input and uses a summarization algorithm to extract key points. It generates the extracted summary as output and shares it with participants, thereby conveying the important points of the meeting to participants.

[1433] Step 7:

[1434] Users input questions through chat during a meeting. The text-based question is received as input and sent to the server. The server analyzes the question and notifies the participants. The analyzed question is obtained as output, and the AI ​​moderator makes notifications based on that. This allows for a smooth exchange of questions and answers.

[1435] Step 8:

[1436] Before the end of the meeting, the server confirms the action items and notifies the participants. It receives as input the action items decided during the meeting and notifies the participants. It outputs a confirmation that the notification was sent, ensuring that the meeting decisions are carried out.

[1437] Step 9:

[1438] After the meeting ends, the server sends a follow-up notification. It receives as input the status of the post-meeting action item, creates and sends a reminder, and as output, provides confirmation that the follow-up notification was sent, ensuring that the action item is followed up appropriately.

[1439] Step 10:

[1440] The server collects and analyzes emotional data, generating appropriate responses and comments according to the participants' emotional state. It receives emotional data such as the user's facial expressions and tone of voice as input, and analyzes it using an emotion analysis engine. It generates appropriate responses and comments based on the analysis results as output, and notifies the participants. This reduces stress for participants and maintains a good meeting atmosphere.

[1441] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1442] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1443] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1444] [Fourth embodiment]

[1445] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1446] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1447] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1448] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1449] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1451] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1452] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1453] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1454] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1455] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1456] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1457] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1458] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting.

[1459] System programs and their processing

[1460] 1. Planning a meeting

[1461] First, the server retrieves the scheduled meeting date and time, the list of attendees, and the agenda. This information can be retrieved from a calendar API or a database. Based on this information, the server sends reminders to attendees. These reminders can be sent via email or a notification system.

[1462] 2. Starting the meeting

[1463] Next, when the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. When the meeting starts, the server launches an AI moderator to manage the progress of the meeting.

[1464] 3. Comment Management

[1465] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[1466] 4. Record and share content

[1467] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[1468] 5. Question and Answer Management

[1469] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[1470] 6. Review and follow up on action items

[1471] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[1472] Specific examples

[1473] Example 1: Planning a meeting

[1474] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday. With this information, the server sends reminders to participants via email, ensuring they don't forget the meeting date.

[1475] Example 2: Speech Management

[1476] During a meeting, a user sends a request to speak from their device. The device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will speak next."

[1477] Example 3: Managing questions and answers

[1478] During a meeting, a user can enter a question via chat. The server receives the question, and the AI ​​moderator responds by saying, "A new question has arrived. Mr. Sato, please answer." This function ensures a smooth flow of questions and answers.

[1479] In this way, the system of the present invention can automate the entire process of an online conference, enabling the conference to proceed efficiently and effectively.

[1480] The processing flow will be explained below.

[1481] Step 1:

[1482] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, either from a calendar API or a database.

[1483] Step 2:

[1484] The server uses the information it retrieves to send reminders to participants, either via email or a notification system.

[1485] Step 3:

[1486] When the meeting starts, the server generates a meeting link, which is a specific URL.

[1487] Step 4:

[1488] The server notifies participants of the generated conference link via email or chat.

[1489] Step 5:

[1490] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[1491] Step 6:

[1492] During a conference, users can send requests to speak through their devices, using buttons or the chat function.

[1493] Step 7:

[1494] The device sends a request to the server, which sends a POST request to the server's API.

[1495] Step 8:

[1496] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[1497] Step 9:

[1498] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[1499] Step 10:

[1500] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[1501] Step 11:

[1502] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[1503] Step 12:

[1504] The server shares the summary with participants via email or cloud storage.

[1505] Step 13:

[1506] During a meeting, a user enters a question in the chat box and presses the send button.

[1507] Step 14:

[1508] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[1509] Step 15:

[1510] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[1511] Step 16:

[1512] The server notifies participants of action items, either via email or through a notification system.

[1513] Step 17:

[1514] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[1515] Example 1

[1516] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1517] Conventional online conference systems require manual management and progress of meetings, which requires a lot of time and effort. Furthermore, there are often ineffective management of the order of comments, recording of discussions, and management of questions and answers during meetings, which reduces the efficiency of meetings. Furthermore, insufficient follow-up after the meeting can lead to delays in the implementation of action items.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1519] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating an AI moderator, receiving speech requests, and organizing the speaking order, means for receiving speech requests from terminals and adding them to a queue, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into a chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, and means for sending follow-up notifications after the end of the meeting. This automates the entire process of online meetings, enabling efficient and effective meeting conduct.

[1520] "Scheduled date and time of the meeting" refers to the specific date and time for which the meeting is scheduled.

[1521] "Participant List" means a list of all participants invited to a Conference.

[1522] An "agenda" is a list of items or topics to be discussed at a meeting.

[1523] "Means for sending reminders" refers to a function that notifies meeting participants so they don't forget about the meeting.

[1524] "Means for generating a meeting link" refers to the function for creating a URL for joining an online meeting.

[1525] "Means for activating AI moderation" refers to the function of activating artificial intelligence that automatically manages the progress of the meeting.

[1526] "Means for receiving a speech request" refers to a function for receiving a request to speak from a participant.

[1527] "Means for organizing the speech order" refers to a function for arranging received speech requests in an appropriate order.

[1528] "Means to add to queue" refers to the ability to add a request to speak to a waiting list.

[1529] "Means of real-time transcription" refers to the ability to record audio during a meeting as text in real time.

[1530] "Means of extracting a summary" refers to the ability to extract key points and important information from the entire content of a meeting.

[1531] "Means for entering questions into chat" refers to the function by which participants enter questions using the chat function.

[1532] "Means for analyzing questions and notifying participants" refers to the function of analyzing the entered questions and notifying the relevant participants at the appropriate time.

[1533] "Means for checking action items" refers to the function for checking specific action items decided in the meeting.

[1534] "Means for sending follow-up notifications" refers to a function for sending follow-up notifications to participants after the end of a conference.

[1535] This invention relates to a system that uses AI to appropriately moderate online meetings and promote communication. This system automates a series of processes, from planning the meeting to proceeding with the meeting, recording and sharing the content, managing questions and answers, and following up after the meeting. Specific embodiments of this system are described below.

[1536] 1. Planning a meeting

[1537] 1.1 Getting the scheduled date, time, attendee list, and agenda

[1538] The server uses a calendar API or database to retrieve the scheduled date and time of the meeting, the attendee list, and the agenda. Specifically, it could use the Google Calendar API or Microsoft Outlook API. This method allows accurate acquisition of basic meeting information.

[1539] 1.2 Sending reminders

[1540] The server sends reminders to participants based on the information obtained, typically using an email sending API (e.g., SendGrid) or push notifications.

[1541] 2. Starting the meeting

[1542] 2.1 Generating and notifying the meeting link

[1543] When the meeting starts, the server generates a meeting link in a specific URL format and notifies the participants. The meeting link is generated using the API of a common online meeting platform (e.g., Zoom Meetings API).

[1544] 2.2 Starting AI moderation

[1545] When the meeting starts, the server launches an AI moderator to manage the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the meeting.

[1546] 3. Comment Management

[1547] 3.1 Sending a request to speak

[1548] Users send requests to speak using the buttons on their devices or the chat function, and this information is sent from the device to the server.

[1549] 3.2 Receiving requests and organizing the order of speaking

[1550] The server receives the requests and organizes the speaking order. The request is added to the queue, and the AI ​​moderator announces, "The next speaker is Mr. / Ms. XX."

[1551] 4. Record and share content

[1552] 4.1 Real-time transcription

[1553] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[1554] 4.2 Abstract Extraction and Notification

[1555] The server extracts a summary of the meeting contents and shares it with participants. It uses a summary generation model (e.g., BART) to extract the main points of the meeting and notifies them via email or other means.

[1556] 5. Question and Answer Management

[1557] 5.1 Entering a Question

[1558] Users can input questions using the chat function on their devices, and this information is sent from the devices to the server.

[1559] 5.2 Question notification and response promotion

[1560] The server receives the question, and the AI ​​moderator prompts the answer at the appropriate time, notifying the user, "There is a new question. Mr. / Ms. XX, please answer."

[1561] 6. Review and follow up on action items

[1562] 6.1 Extracting Action Items

[1563] Before the end of the meeting, the server checks the action items and extracts any uncompleted tasks or action targets.

[1564] 6.2 Follow-up Notifications

[1565] The server will send periodic follow-up notifications after the meeting to help ensure action items are carried out.

[1566] Specific examples

[1567] Example 1: Send a reminder

[1568] The server retrieves meeting information scheduled for "October 15, 2023 at 2:00 PM" from the Google Calendar API and sends reminders to participants using the SendGrid API.

[1569] Example 2: Speech Management

[1570] During a meeting, when a user clicks a button on their device to send a request to speak, the server receives the request and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[1571] Example 3: Managing questions and answers

[1572] During a meeting, if a user types in chat, "More details about this topic?", the server receives the question and the AI ​​moderator notifies them, "There is a new question. Mr. Sato, please answer."

[1573] Prompt Sentence Examples

[1574] Please tell me more about how to notify "X-Person will be speaking next."

[1575] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1576] Step 1: Get the scheduled meeting date, time, attendee list, and agenda

[1577] The server retrieves the scheduled date and time of the meeting, the attendee list, and the agenda from a calendar API (e.g., Google Calendar API) or a database. In this step, the server confirms the meeting date, attendees, and agenda based on the data retrieved through the API. The input is information from the calendar API or database, and the output is the meeting information.

[1578] Specific behavior:

[1579] The server sends a request to the Google Calendar API to retrieve the meeting information.

[1580] The server reads the participant list from a database.

[1581] Step 2: Send a reminder

[1582] The server sends reminders based on the acquired meeting information. This process uses an email sending API (e.g., SendGrid) or a push notification service to notify participants. The input is the meeting information acquired in step 1, and the output is the sent reminder notification.

[1583] Specific behavior:

[1584] The server uses the SendGrid API to send a reminder to participants saying, "We're planning a meeting for October 15th."

[1585] The server uses a push notification service to send notifications to participants' devices.

[1586] Step 3: Generate and share the meeting link

[1587] When the meeting start time arrives, the server uses the API of the online meeting platform (e.g., Zoom Meetings API) to generate a meeting link and notify participants. The input is the meeting start time and participant information, and the output is the generated meeting link and its notification.

[1588] Specific behavior:

[1589] The server uses the Zoom Meetings API to generate the meeting link.

[1590] The server will send the generated link to participants via email and push notification.

[1591] Step 4: Start the AI ​​moderator

[1592] The server starts the AI ​​moderator as soon as the meeting starts and manages the progress of the meeting. Specifically, it uses an AI model (e.g., OpenAI GPT-3) to support the progress of the meeting. The input is the trigger to start the meeting, and the output is the startup status of the AI ​​moderator.

[1593] Specific behavior:

[1594] The server launches OpenAI GPT-3 at the start of the meeting and begins moderating.

[1595] Step 5: Sending and receiving requests to speak

[1596] Users send speech requests using the buttons or chat function on their devices, and the devices send this information to the server. The server receives the requests and organizes the speech order. The input is the speech request from the user, and the output is the organized speech order.

[1597] Specific behavior:

[1598] The user clicks the "Request to speak" button on the device.

[1599] The terminal sends a request to speak to the server.

[1600] The server receives the request and adds it to a queue.

[1601] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. XX."

[1602] Step 6: Real-time transcription

[1603] The server transcribes what is said during the meeting in real time using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is the audio data during the meeting, and the output is the transcribed text.

[1604] Specific behavior:

[1605] The server sends the audio data to the Google Cloud Speech-to-Text API, which returns it as text.

[1606] Transcribe what is being said in real time.

[1607] Step 7: Extract and notify the summary

[1608] The server extracts a summary of the meeting content and notifies participants. The summary is created using a summary generation model (e.g., BART). The input is the transcribed text, and the output is the summarized text.

[1609] Specific behavior:

[1610] The server uses the BART model to extract summaries from the transcribed content.

[1611] The server will notify participants of the summary via email.

[1612] Step 8: Manage questions and answers

[1613] Users input questions into the chat through their devices, the server receives and analyzes the questions, and the AI ​​moderator facilitates the answers. The input is the user's question, and the output is the analyzed question and answer instructions.

[1614] Specific behavior:

[1615] The user types a question into the chat box.

[1616] The terminal sends a query to the server.

[1617] The server notifies the AI ​​moderator of the question and prompts them, "There is a new question. Mr. / Ms. XX, please answer."

[1618] Step 9: Review and follow up on action items

[1619] The server checks action items before the end of the meeting, extracts uncompleted tasks and action targets, and notifies participants. It also periodically sends follow-up notifications after the meeting ends. The input is the action items recorded during the meeting, and the output is confirmation and follow-up notifications.

[1620] Specific behavior:

[1621] The server checks the task list for the meeting and extracts any uncompleted items.

[1622] The server will send follow-up notifications one week after the meeting ends.

[1623] (Application example 1)

[1624] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1625] Currently, maintaining efficiency and consistency in online meetings and virtual events is difficult, particularly in managing participants' opinions and smoothly managing the flow of real-time questions and answers. Furthermore, in environments with a large number of participants, such as online product briefings and campaign events held in virtual stores, managing speaking requests and questions becomes even more cumbersome, hindering effective communication.

[1626] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1627] In this invention, the server includes means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, and means for managing speech requests and questions. This enables efficient and consistent management of the flow of speech requests and questions at online product briefings and campaign events in virtual stores, enabling effective communication.

[1628] The "scheduled date and time of the meeting" refers to the date and time when the meeting will start.

[1629] A "participant list" refers to a list of people who plan to attend a meeting or event.

[1630] An "agenda" is a list of topics or items to be discussed at a meeting or event.

[1631] "Means for sending reminders" means a function or system for sending advance notice of a meeting or event to participants.

[1632] "Conference Link" means a specific web address or invitation link through which participants can join a conference or event.

[1633] "Means of notification" refers to a communication means for conveying information such as a conference link to participants.

[1634] "AI moderation" refers to a system that uses artificial intelligence to automatically manage the progress of meetings and events.

[1635] "Request to speak" means a request by a participant for permission to speak during a meeting or event.

[1636] "Means for organizing the order of speech" refers to a system or function that accepts speech requests from participants and arranges the speech in the appropriate order.

[1637] "Means of real-time transcription" refers to functions or systems that convert statements made during meetings or events into text data in real time.

[1638] "Means to extract summaries and share them with participants" refers to a function for summarizing the contents of a meeting or event and conveying the main points to participants.

[1639] "Means for analyzing questions and notifying participants" refers to a system or function for analyzing questions entered in chat and informing the appropriate participants of their content.

[1640] "Means to review action items and notify participants" refers to the ability to review outstanding tasks and action items before the end of a meeting or event and notify participants as needed.

[1641] "Means for sending follow-up notifications" refers to a system or function for checking the progress of action items after a meeting or event and sending follow-up notifications to participants.

[1642] "Virtual store" refers to a virtual store that operates on the Internet, an online sales and service platform that does not have an actual physical location.

[1643] An "online product briefing" refers to a product briefing event held over the Internet.

[1644] "Campaign Event" means a series of online events conducted to promote a particular product or service.

[1645] "Means for managing speech requests and questions" refers to a system or function for managing speech requests and questions from participants during an event in a virtual store.

[1646] This invention relates to a system and method for smoothly conducting online product briefings and campaign events in a virtual store. This system uses AI to automate the event moderation and provides functions for managing participants' comments and smoothly managing the flow of questions and answers. The implementation of this system includes the following specific configuration and operation procedures.

[1647] Hardware and software used

[1648] server:

[1649] Role: Overall moderator, sending reminders, generating meeting links, managing speaking requests and questions, real-time transcription, extracting summaries, and following up notifications.

[1650] Examples: Google Cloud Platform, AWS (Amazon Web Services)

[1651] Device:

[1652] Role: A device for participants to send requests to speak and type questions through chat.

[1653] Examples: smartphones, tablets, and PCs

[1654] Server processing

[1655] 1. Send reminders:

[1656] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda, and sends reminders to participants, allowing them to be aware of the meeting or event in advance.

[1657] Example: A server sends reminder emails to participants on the weekend about next week's event.

[1658] 2. Conference link generation and notification:

[1659] At the start time of the meeting, the server generates a meeting link and notifies participants, who can use this link to join the online event.

[1660] Example: Before the event starts, a meeting link is generated and sent to participants' email addresses.

[1661] 3. Managing requests and questions:

[1662] Participants can send requests to speak through their devices, and the server adds them to a queue and organizes them in the appropriate order.

[1663] Additionally, when a question is sent using the chat function, the server analyzes it and notifies the appropriate participant.

[1664] Example: During an event, a participant sends a request to speak, and the server organizes the order and notifies them, "Person A will speak next." Also, a question is sent via chat, and the server notifies them, "There is a new question. Person B, please answer."

[1665] 4. Real-time transcription and summary extraction:

[1666] The server transcribes speeches made during the event in real time, and a summarization algorithm is used to extract key points and share them with participants.

[1667] Example: After the event, a transcript and summary of what was said is automatically emailed to attendees.

[1668] 5. Follow-up Notification:

[1669] Before the event ends, the server checks for action items, extracts any incomplete tasks and action items, and notifies participants. After the event ends, the server periodically sends follow-up notifications to ensure that action items are carried out.

[1670] Example: After the event ends, the server notifies participants that "Uncompleted task: Create explanatory materials about the product details."

[1671] Examples of prompt statements

[1672] Implement a function send_reminder to send reminders to participants. It uses Gmail's SMTP server and takes the specified email address, subject, and body as arguments. It also requires user authentication to send the email.

[1673] Through the above process, the present invention can realize effective communication and smooth event progress within the virtual store.

[1674] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1675] Step 1:

[1676] The server obtains the scheduled date and time of the meeting, the participant list, and the agenda. This information is obtained as input from a database or calendar API. The output is the obtained scheduled date and time, participant list, and agenda data. Specifically, the server calls the API to extract the data and saves it in an internal database.

[1677] Step 2:

[1678] The server sends reminders to participants based on the information it obtains. The input is the scheduled date and time of the meeting, the participant list, and the agenda. The output is a reminder email or notification sent to participants. Specifically, the server generates the reminder content and sends the email using an SMTP server.

[1679] Step 3:

[1680] At the start time of the conference, the server generates a conference link and notifies the participants. The input is the start time and the participant list. The output is the generated conference link and an email notifying the participants. Specifically, the server generates a conference URL and sends a notification via email.

[1681] Step 4:

[1682] A user uses a terminal to send a speech request. The input is the user's request information. The output is adding the request information to the speech queue by sending it to the server. In concrete terms, the user inputs a request through the interface of the terminal, and the terminal sends the information to the server.

[1683] Step 5:

[1684] The server receives speech requests and organizes the speech order. The input is request information from the user. The output is an organized speech order list. Specifically, the server adds requests to a queue and manages the requests in the appropriate order.

[1685] Step 6:

[1686] A user inputs a question into the chat, and the terminal sends the question to the server. The input is the question entered by the user. The output is the question being sent to the server. In concrete terms, the user inputs a question into the chat interface and presses the send button.

[1687] Step 7:

[1688] The server analyzes the question and notifies the appropriate participant. The input is the question sent by the user via chat. The output is data used to analyze and notify the question. Specifically, the server uses natural language processing to analyze the question and notify the appropriate respondent.

[1689] Step 8:

[1690] The server transcribes the meeting content in real time and extracts a summary. The input is the audio data of the meeting. The output is the transcribed text data and summary data. Specifically, the server uses a speech recognition engine to convert the audio data into text and then summarizes the text using a summarization algorithm.

[1691] Step 9:

[1692] The server checks action items and notifies participants before the end of the meeting. The input is the list of action items recorded during the meeting. The output is to notify participants of incomplete action items. Specifically, the server scans the action item list and adds incomplete tasks to the notification list.

[1693] Step 10:

[1694] The server sends follow-up notifications after the meeting ends. The input is the progress of the action items at the time the meeting ends. The output is a follow-up notification email or notification. Specifically, the server checks the status of the action items and periodically sends follow-up notifications.

[1695] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1696] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and further combines it with an emotion engine that recognizes user emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[1697] System programs and their processing

[1698] 1. Planning a meeting

[1699] First, the server retrieves the scheduled meeting date and time, the attendee list, and the agenda. This information can be retrieved from a calendar API or a database. Then, based on this information, the server sends reminders to the attendees. Reminders can be sent via email or a notification system.

[1700] 2. Starting the meeting

[1701] When the meeting starts, the server generates a meeting link and notifies the participants. The meeting link is a specific URL that participants can use to join the meeting. The server then launches an AI moderator to manage the meeting.

[1702] 3. Comment Management

[1703] Users send requests to speak through their devices. Speak requests are made using the buttons or chat function on the device. The device then sends the request to the server, which receives the request and organizes the speaking order. The AI ​​moderator supports a smooth process by notifying users, "The next speaker is Mr. / Ms. X."

[1704] 4. Record and share content

[1705] The server transcribes what is said during the meeting in real time. The server performs this process using a speech recognition engine. A summary of the meeting content is also extracted, and important points are shared with participants. This allows all participants to accurately understand the content of the meeting.

[1706] 5. Question and Answer Management

[1707] During the meeting, users can enter questions into the chat via their devices. The questions entered into the chat are received by the server, and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer." This function ensures a smooth exchange of questions and answers.

[1708] 6. Review and follow up on action items

[1709] Before the end of the meeting, the server checks the action items, extracts any uncompleted tasks or action targets, and notifies the participants. After the meeting ends, the server periodically sends follow-up notifications to ensure that the action items are carried out.

[1710] 7. Emotion Engine Functions

[1711] By incorporating an emotion engine, the server can analyze the user's emotional state in real time during a meeting. The emotion engine recognizes emotions from the user's facial expressions, tone of voice, and speech content, and analyzes and stores these as data.

[1712] The AI ​​moderator can generate appropriate responses and comments based on this emotional data, reducing stress for participants and maintaining a good meeting atmosphere. For example, if a user is nervous, the AI ​​moderator will say, "Please relax and speak."

[1713] Specific examples

[1714] Example 1: Planning a meeting and sending reminders

[1715] The server retrieves the scheduled date, time, participant list, and agenda for the upcoming meeting on Monday and sends reminders to participants via email, reminding them to remember the meeting date.

[1716] Example 2: Speech management and emotion recognition

[1717] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to the queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next." At the same time, the emotion engine analyzes the user's emotional state, and if they appear tense, it will say, "Please relax and speak."

[1718] Example 3: Question and answer management and sentiment analysis

[1719] During a meeting, a user types a question into the chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Sato-san, please answer." At the same time, the emotion engine analyzes Sato-san's anxious expression and provides additional support, such as, "If the question is complicated, please speak more slowly."

[1720] In this way, the system of the present invention not only automates the entire process of online meetings, enabling efficient and effective meeting conduct, but also supports more human and interactive meetings through the introduction of an emotion engine.

[1721] The processing flow will be explained below.

[1722] Step 1:

[1723] The server retrieves the scheduled meeting date and time, the attendee list, and the agenda, which can be retrieved from a calendar API or a database.

[1724] Step 2:

[1725] The server uses the information it retrieves to send reminders to participants, which can be sent via email or a notification system.

[1726] Step 3:

[1727] When the meeting starts, the server generates a meeting link, which is a specific URL.

[1728] Step 4:

[1729] The server notifies participants of the generated conference link via email or chat.

[1730] Step 5:

[1731] When a meeting starts, the server launches an AI moderator, which automatically manages the progress of the meeting.

[1732] Step 6:

[1733] Users send requests to speak through their devices, using buttons or the chat function.

[1734] Step 7:

[1735] The device sends a request to the server, which sends a POST request to the server's API.

[1736] Step 8:

[1737] The server receives requests to speak and organizes the order in which they should be spoken. Requests are added to a queue and then organized according to an ordering algorithm.

[1738] Step 9:

[1739] The AI ​​moderator will announce, "The next speaker will be Mr. / Ms. X," in real time via voice synthesis and text display.

[1740] Step 10:

[1741] The server transcribes the meeting in real time, using a speech recognition engine to convert what is being said into text.

[1742] Step 11:

[1743] The server records the meeting content and extracts a summary, using a summarization algorithm to extract key points.

[1744] Step 12:

[1745] The server shares the summary with participants via email or cloud storage.

[1746] Step 13:

[1747] During a meeting, a user enters a question in the chat box and presses the send button.

[1748] Step 14:

[1749] The server receives the question and the AI ​​moderator notifies the user in a timely manner, saying, "There is a new question. Mr. / Ms. X, please answer."

[1750] Step 15:

[1751] During the meeting, the server uses an emotion engine to analyze the user's emotional state by detecting facial expressions, tone of voice, and speech content and analyzing the data.

[1752] Step 16:

[1753] The emotion engine analyzes the user's emotional state and sends the results back to the server. The emotional data is then passed on to the AI ​​host.

[1754] Step 17:

[1755] Based on the emotional data, the AI ​​moderator generates appropriate comments, such as "Please relax and speak," and notifies the speaker in real time through voice synthesis and text display.

[1756] Step 18:

[1757] Before the end of the meeting, the server checks for action items, parses the log, and extracts any outstanding tasks.

[1758] Step 19:

[1759] The server notifies participants of action items, either via email or through a notification system.

[1760] Step 20:

[1761] After the meeting ends, the server periodically sends follow-up notifications, and a scheduled job runs to report any outstanding action items.

[1762] Example 2

[1763] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1764] Ensuring smooth online meetings and smooth communication between participants requires a lot of manual work. This includes planning meetings, sending reminders, managing comments, recording and sharing content, coordinating questions and answers, and following up after the meeting. It is also important to understand participants' emotional states and communicate appropriately. However, performing these tasks manually is labor-intensive and inefficient. Furthermore, analyzing participants' emotions in real time and responding appropriately is practically difficult. Therefore, there is a need for an automated system to solve these issues and improve the efficiency and quality of online meetings.

[1765] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1766] In this invention, the server includes: means for acquiring the scheduled date and time of the meeting, a participant list, and an agenda; means for sending reminders to participants; means for generating a meeting link at the start time of the meeting and notifying participants; means for activating an AI moderator, receiving speaking requests, and organizing the speaking order; means for transcribing the meeting content in real time; means for extracting a summary of the meeting content and sharing it with participants; means for inputting questions into a chat, analyzing the questions, and notifying participants; means for checking action items before the end of the meeting and notifying participants; means for sending follow-up notifications after the end of the meeting; and means for analyzing user emotions in real time using an emotion engine and generating appropriate responses and comments. This automates the entire process of online meetings, enabling efficient and effective meeting conduct and also realizing smoother communication by understanding participants' emotions in real time and responding appropriately.

[1767] The "scheduled date and time of the conference" refers to the specific date and time when the conference is scheduled to start.

[1768] A "participant list" is a list that includes information such as names and contact details of people who will be attending a conference.

[1769] An agenda is a list of the main topics and subjects of a meeting that is compiled in advance and used as a guideline for proceeding.

[1770] A "reminder" is an email or message that notifies participants in advance so they don't forget about the meeting schedule.

[1771] "Meeting Link" means a specific URL used by an online meeting platform that allows participants to join a meeting by clicking on the link.

[1772] "AI moderation" is a system that uses artificial intelligence to manage the progress of a meeting, automatically coordinating remarks and managing questions.

[1773] A "request to speak" is a request made by a participant who wishes to speak during a conference to inform the moderator of that request.

[1774] "Speaking order" refers to the order in which people speak in a meeting, and is adjusted to ensure smooth progress.

[1775] "Transcription" is the process or result of converting audio from a meeting into text in real time.

[1776] A "summary" is a concise summary of the main points and conclusions discussed at a meeting that is shared with all participants.

[1777] "Chat" is a real-time means of communication using text messages.

[1778] "Action items" refer to specific actions or tasks decided during a meeting.

[1779] "Follow-up notifications" are reminders sent to participants after the meeting to ensure action items are carried out.

[1780] An "emotion engine" is a system that analyzes a user's facial expressions, tone of voice, speech content, etc. to recognize their emotional state and generate an appropriate response.

[1781] "Appropriate responses and comments" are statements and feedback generated by AI based on the analyzed emotional state, designed to reduce participants' stress and create a positive atmosphere in the meeting.

[1782] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication. It also incorporates an emotion engine that recognizes the user's emotions. This system automates a series of processes, from planning the meeting, conducting the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[1783] First, the server retrieves the scheduled meeting date and time, the list of participants, and the agenda. This information can be retrieved from a calendar API, such as the Google Calendar API or the Microsoft Exchange API, or from an internal database. Based on this information, the server generates and sends reminders to participants via email or a notification system (for example, Slack or Microsoft Teams).

[1784] Next, when the meeting's start time approaches, the server generates a meeting link to be used in an online meeting platform (such as Zoom, Microsoft Teams, or Google Meet) and notifies participants via email or chat. At the same time, the server launches an AI moderator program to manage the meeting.

[1785] During a meeting, users use their devices (PC, tablet, smartphone) to send requests to speak. This request is sent using the device's buttons or chat function, and the device then sends it to the server. The server receives the requests and organizes the speaking order. The AI ​​moderator notifies participants, such as "Person XX will be speaking next," helping to ensure smooth progress.

[1786] The server transcribes what is said during the meeting in real time using engines such as Google Cloud Speech-to-Text, IBM Watson Speech to Text, and Microsoft Azure Cognitive Services for speech recognition. The server then summarizes the meeting, extracts key points, and shares them with participants via email or in a document format.

[1787] During the meeting, users can use the chat function to enter questions. The questions entered in the chat are received by the server, and the AI ​​moderator notifies them by saying, "There is a new question. Mr. / Ms. X, please answer."

[1788] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. After the meeting ends, the server periodically sends follow-up notifications to help ensure that action items are carried out. This follow-up is achieved by using reminder emails and integration with task management tools (Trello and Asana).

[1789] The system also incorporates an emotion engine. The server uses an emotion engine (such as Microsoft Azure Emotion API, Amazon Rekognition, or Google Cloud AI) to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the analysis results, the AI ​​moderator generates appropriate responses and comments to reduce user stress and maintain a good meeting atmosphere. For example, it might say to a nervous user, "Please relax and speak."

[1790] Specific examples

[1791] Meeting planning and reminders

[1792] The server calls the Google Calendar API to retrieve meeting information for the next week, and automatically sends reminders to all participants via Gmail based on the meeting date, time, and participant list.

[1793] Meeting link generation and notification

[1794] 15 minutes before the meeting starts, the server creates a new meeting using the Zoom API and generates a meeting link, which is then sent to participants via Slack.

[1795] Organizing requests to speak

[1796] The user clicks the "Request to speak" button on their PC. The device sends this request to the server, which adds it to the queue. The AI ​​moderator announces, "Tanaka will be speaking next." At the same time, the emotion engine analyzes Tanaka's facial expressions and, if she appears nervous, says, "Please relax and speak."

[1797] Managing Questions and Answers

[1798] The user types in the chat box, "When is this assignment due?" The server receives this message, and the AI ​​moderator announces, "There is a new question. Sato-san, please answer."

[1799] Prompt Sentence Examples

[1800] "How should I address a participant who seems nervous during a remote meeting?"

[1801] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1802] Step 1:

[1803] Meeting planning

[1804] The server retrieves the scheduled date and time of the meeting, the participant list, and the agenda. This is done by using the Google Calendar API, Microsoft Exchange API, and also by reading data from an internal database. Specifically, it sends a request to the API endpoint and parses the data returned as a response to retrieve the meeting information. This information is the input data, and the retrieved date and time of the meeting, the participant list, and the agenda are the output data.

[1805] Step 2:

[1806] Send a reminder

[1807] The server generates reminders based on the scheduled date and time, participant list, and agenda of the meeting. These reminders are sent to participants via email or notification system via Gmail API or Slack API. The input data is the retrieved meeting information, and the output data is the reminder sending status.

[1808] Step 3:

[1809] Generate a Conference Link

[1810] When the meeting start time approaches, the server generates a new meeting link using the Zoom API, Microsoft Teams API, or Google Meet API. It sends an appropriate request to the API and obtains the meeting link as a response. The input data is the meeting start time, and the output data is the generated meeting link.

[1811] Step 4:

[1812] Meeting link notification

[1813] The server notifies participants of the generated meeting link via email or chat tool. The Slack API is used to notify all participants of the link. The input data is the generated meeting link, and the output data is the notification sending status.

[1814] Step 5:

[1815] AI moderator activation

[1816] The server starts an AI moderator program at the same time as the start of the meeting. The moderator program manages the progress of the meeting using a generative AI model (e.g., GPT-3). The input data is the start signal for the meeting, and the output data is the start status of the AI ​​moderator program.

[1817] Step 6:

[1818] Sending a request to speak

[1819] A user uses a terminal to send a speech request, and the terminal sends this request to the server. The input data is the user's speech request, and the output data is the request status sent to the server.

[1820] Step 7:

[1821] Receiving and organizing requests to speak

[1822] The server organizes speech requests received from terminals in order, adds the requests to a queue, and selects the next user to speak. The input data is the received speech requests, and the output data is the organized speech order.

[1823] Step 8:

[1824] Speaking order notification

[1825] The server notifies the AI ​​moderator, "The next speaker is Mr. / Ms. X." The input data is the organized order of speeches, and the output data is the notification sending status.

[1826] Step 9:

[1827] Meeting transcription

[1828] The server transcribes the meeting content in real time using a speech recognition engine such as Google Cloud Speech-to-Text or IBM Watson Speech to Text. The input data is the meeting audio, and the output data is the transcribed text.

[1829] Step 10:

[1830] Summarizing and sharing meeting content

[1831] The server processes the transcribed text with a summarization algorithm to extract key points, and shares the extracted summary with participants via email or document format. The input data is the transcribed text, and the output data is the summarized text and the sharing status.

[1832] Step 11:

[1833] Receiving and parsing questions

[1834] The user enters a question using the chat function. The server receives the question, and the AI ​​moderator analyzes the question and notifies the person responsible for answering it. The input data is the question entered in the chat, and the output data is the analysis result and notification status.

[1835] Step 12:

[1836] Action item confirmation and notifications

[1837] Before the end of the meeting, the server checks for action items (uncompleted tasks and action targets) and notifies participants. The input data is the meeting summary and task list, and the output data is the notification status.

[1838] Step 13:

[1839] Sending follow-up notifications

[1840] Even after the meeting ends, the server periodically sends follow-up notifications. It uses reminder emails and task management tools to ensure that action items are carried out. The input data is a list of action items, and the output data is the sending status of the follow-up notifications.

[1841] Step 14:

[1842] Emotion engine analysis and response generation

[1843] The server uses an emotion engine to analyze the user's facial expressions, voice tone, and speech content during the meeting. Based on the results of this analysis, the AI ​​moderator generates appropriate responses and comments. For example, if the server detects that the user is nervous, it generates a message such as "Please relax and speak." The input data is the user's emotional state, and the output data is the response message.

[1844] (Application example 2)

[1845] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1846] With conventional online conference systems, planning, proceeding with the meeting, recording and sharing the contents, managing questions and answers, and following up after the meeting are all done manually, making it difficult to manage efficiently. Furthermore, the meeting cannot proceed taking into account the emotional state of the participants, resulting in insufficient communication. This results in issues such as meetings not proceeding smoothly and participants' opinions not being fully elicited.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the scheduled date and time of the meeting, the participant list, and the agenda, means for sending reminders to participants, means for generating a meeting link at the start time of the meeting and notifying the participants, means for activating AI moderation, receiving speech requests, and organizing the speaking order, means for transcribing the meeting content in real time, means for extracting a summary of the meeting content and sharing it with participants, means for inputting questions into the chat, analyzing the questions, and notifying the participants, means for checking action items before the end of the meeting and notifying the participants, means for sending follow-up notifications after the end of the meeting, and means for collecting and analyzing emotional data and generating appropriate responses and comments according to the emotional states of the participants. This not only automates the entire process of online meetings, but also enables appropriate responses according to the emotional states of the participants, thereby achieving efficient and effective meeting progress.

[1848] The "scheduled date and time of the meeting" refers to the specific date and time when the meeting will start.

[1849] The "participant list" indicates a list of users who are scheduled to participate in the conference.

[1850] An "agenda" is a list of specific topics or issues to be discussed at a meeting.

[1851] "Means for sending reminders" refers to a method or system for notifying users of upcoming meetings.

[1852] "Conference link" refers to the URL and connection information required for a user to participate in a conference.

[1853] "AI moderation" refers to a system that uses artificial intelligence to manage the progress of online meetings and call on speakers at appropriate times.

[1854] "Means for receiving requests to speak and organizing the order in which they speak" refers to a method or system for receiving requests to speak from participants and managing them in an appropriate order.

[1855] "Means for transcribing meeting content in real time" refers to technology that instantly converts audio during a meeting into text.

[1856] "Means for extracting a summary of the meeting content and sharing it with participants" refers to a method or system for summarizing the important points and main points from the meeting and communicating them to participants.

[1857] "Means of entering questions into chat, analyzing the questions, and notifying participants" refers to a method of receiving text questions from participants, understanding their content, and notifying participants at an appropriate time.

[1858] An "action item" refers to a specific task or work decided during a meeting.

[1859] "Means for sending follow-up notifications" refers to a method or system for checking whether action items are being properly carried out after the meeting and sending reminders as necessary.

[1860] "Emotion data" refers to information about the user's emotional state that can be inferred from facial expressions, tone of voice, and the like.

[1861] "Means for generating appropriate responses and comments" refers to methods and systems for providing necessary support and encouragement to participants based on emotional data.

[1862] This invention relates to a system that uses AI to appropriately moderate online meetings and stimulate communication, and combines it with an emotion engine that recognizes the user's emotions. This system can automate a series of processes, from planning the meeting, proceeding with the meeting, recording and sharing the contents, managing questions and answers, following up after the meeting, and recognizing the user's emotions.

[1863] System programs and their processing

[1864] Hardware and Software

[1865] Hardware: Servers, devices (smartphones, PCs, etc.)

[1866] Software: SQLite (database), smtplib (Python library), email.mime (Python library), speech recognition engine (e.g., Google Speech-to-Text API), sentiment analysis engine (e.g., Amazon Comprehend)

[1867] Meeting planning

[1868] The server retrieves the scheduled meeting date and time, the participant list and the agenda from the database and sends reminders to participants, which are sent via email.

[1869] Specific examples

[1870] For a meeting scheduled for next Monday, the server retrieves the scheduled date and time, the attendee list, and the agenda from the database, and sends email reminders to the attendees, helping them remember to plan for the meeting.

[1871] Example prompt sentence:

[1872] "Get the date, time, participant list, and agenda for your next seminar and send reminder emails to each participant."

[1873] Starting a meeting

[1874] When the meeting starts, the server generates a meeting link and notifies the participants, who can then join the meeting using the generated URL.

[1875] Speech Management

[1876] Users send requests to speak through their devices, which are received by the server and sorted in the appropriate order. The AI ​​moderator announces the order of speaking and helps ensure the meeting proceeds smoothly.

[1877] Specific examples

[1878] During a meeting, when a user sends a request to speak from their device, the device sends the request to the server, which receives the request and adds it to a queue. The server then sorts the requests in the appropriate order, and the AI ​​moderator announces, "Mr. Tanaka will be speaking next."

[1879] Record and share meeting content

[1880] The server transcribes what is said during the meeting in real time, and a summary is extracted using a summarization algorithm and shared with participants, allowing all participants to accurately understand the content of the meeting.

[1881] Managing Questions and Answers

[1882] During the meeting, users can enter questions into the chat through their devices. The questions are received by the server, and the AI ​​moderator notifies them at the appropriate time. This function allows questions and answers to be exchanged smoothly.

[1883] Specific examples

[1884] During a meeting, a user enters a question via chat, the server receives the question, and the AI ​​moderator prompts, "A new question has arrived. Mr. Sato, please answer."

[1885] Example prompt sentence:

[1886] "Receive chat questions and notify us at the right time"

[1887] Review and follow up on action items

[1888] The server checks for action items and notifies participants before the end of the meeting, and also sends follow-up notifications after the meeting to ensure that action items are carried out.

[1889] Emotion Engine Functions

[1890] By incorporating an emotion engine, the server analyzes the user's emotional state in real time during the meeting, allowing the AI ​​moderator to reduce participants' stress and maintain a good meeting atmosphere.

[1891] Specific examples

[1892] During a meeting, the emotion engine analyzes the user's level of tension from their facial expressions and tone of voice, and the AI ​​moderator encourages them to "relax and speak."

[1893] As described above, by using the above-mentioned hardware and software, the online conference system of the present invention can automate the entire conference process and respond appropriately to the emotional state of the participants, thereby realizing efficient and effective conference proceedings.

[1894] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1895] Step 1:

[1896] The server retrieves the scheduled date and time, participant list, and agenda of the meeting from the database. It receives a meeting ID as input and searches the database for the corresponding scheduled date and time, participant list, and agenda of the meeting and outputs it. This data is used as information required for subsequent processing.

[1897] Step 2:

[1898] The server sends reminders to participants based on the information it has obtained. It receives the participant list, scheduled date and time, and agenda as input, creates and sends an email, and receives a confirmation that the reminder was sent as output. The reminder helps participants remember that the meeting is happening.

[1899] Step 3:

[1900] When the meeting start time arrives, the server generates a meeting link and notifies the participants. It receives the meeting ID as input and generates a meeting link (URL). The generated URL is sent as output to the participants via email. This prepares the participants to join the meeting.

[1901] Step 4:

[1902] The server starts the AI ​​moderator, receives speaking requests, and organizes the speaking order. It receives speaking requests as input and organizes the speaking order. It adds the requests to the queue in the appropriate order. It outputs organized speaking order information. This allows the meeting to proceed smoothly.

[1903] Step 5:

[1904] The server transcribes what is said during the meeting in real time. It receives the voice data from the meeting as input and converts it into text data using a speech recognition engine. It obtains the text data generated in real time as output. This records the contents of the meeting.

[1905] Step 6:

[1906] The server extracts a summary of the meeting content and shares it with participants. It receives the transcribed text data as input and uses a summarization algorithm to extract key points. It generates the extracted summary as output and shares it with participants, thereby conveying the important points of the meeting to participants.

[1907] Step 7:

[1908] Users input questions through chat during a meeting. The text-based question is received as input and sent to the server. The server analyzes the question and notifies the participants. The analyzed question is obtained as output, and the AI ​​moderator makes notifications based on that. This allows for a smooth exchange of questions and answers.

[1909] Step 8:

[1910] Before the end of the meeting, the server confirms the action items and notifies the participants. It receives as input the action items decided during the meeting and notifies the participants. It outputs a confirmation that the notification was sent, ensuring that the meeting decisions are carried out.

[1911] Step 9:

[1912] After the meeting ends, the server sends a follow-up notification. It receives as input the status of the post-meeting action item, creates and sends a reminder, and as output, provides confirmation that the follow-up notification was sent, ensuring that the action item is followed up appropriately.

[1913] Step 10:

[1914] The server collects and analyzes emotional data, generating appropriate responses and comments according to the participants' emotional state. It receives emotional data such as the user's facial expressions and tone of voice as input, and analyzes it using an emotion analysis engine. It generates appropriate responses and comments based on the analysis results as output, and notifies the participants. This reduces stress for participants and maintains a good meeting atmosphere.

[1915] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1916] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inferenc...

Claims

1. means for obtaining the scheduled date and time of the meeting, a list of participants, and an agenda; a means of sending reminders to participants; A means for generating a conference link at the start time of the conference and notifying participants; A means to launch the AI ​​moderator, receive speaking requests, and organize speaking order; A means to transcribe meeting content in real time, A means to extract a summary of the meeting content and share it with participants, A means for inputting a question into the chat, analyzing the question, and notifying participants of the result; A way to review and notify participants of action items before the meeting ends, and A means to send follow-up notifications after the meeting has concluded; A system including:

2. 10. The system of claim 1, wherein requests to speak are received, added to a queue, and organized in an appropriate order.

3. The system of claim 1 , wherein the meeting content is transcribed by a speech recognition engine and a summary is extracted using a summarization algorithm.

Images