system

Abstract

We provide the system. [Solution] A method for obtaining the activity schedules of multiple users and calculating common free time, A method for automatically booking a meeting place based on available time, A means of using speech recognition to transcribe speech during a meeting into text and to monitor its progress, A method for automatically generating meeting minutes and distributing them to users, A means of collecting feedback from users about the meeting and using it to improve the next meeting, A means of displaying the progress of the agenda in real time using a visual aid device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In many modern organizations, a huge amount of time and labor are spent on meeting coordination, progress, and post-processing. In particular, scheduling adjustments for participants, securing a meeting venue, and creating meeting minutes after the meeting are factors that reduce business efficiency. These tasks are often performed manually, and especially the younger generation tends to be taken a lot of time for these tasks due to lack of experience. Therefore, it is important to automate and improve these processes related to meetings.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes means for acquiring the schedules of multiple users and calculating common free time, means for automatically reserving a meeting place based on the free time, means for transcribing speech during a meeting into text using speech recognition and monitoring the progress, means for automatically generating meeting minutes and distributing them to users, and means for collecting feedback from users on the meeting to help improve future meetings. This makes it possible to improve the efficiency of meeting-related tasks and reduce the time burden on participants.

[0006] "User" refers to an individual or group that uses the system and is the entity that provides input such as schedule information and feedback.

[0007] "Schedules" refer to information indicating activities or tasks that a user plans to perform at a specific date and time in the future, and are stored in a calendar system or similar.

[0008] "Free time" refers to time slots when multiple users have no prior appointments, and these can be used as potential meeting times.

[0009] A "meeting place" refers to a physical or virtual location, a space used by participants to conduct a meeting.

[0010] "Speech recognition" refers to a technology that processes audio signals and converts them into text format, and is used to record what is said during meetings.

[0011] "Transcription" refers to the process of converting information such as audio into text data, and is used for creating meeting minutes.

[0012] An "agenda" refers to a list of themes or topics to be discussed in a meeting, and is used to guide the progress of the meeting.

[0013] "Meeting minutes" refers to a document that summarizes what was discussed, decided, and action items taken during a meeting, and is distributed to participants after the meeting.

[0014] "Feedback" refers to the opinions and evaluations that users provide about the content, flow, and overall experience of a meeting, which can be used to improve future meetings. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system aimed at improving the efficiency of meetings, which automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress during meetings, and creation and distribution of meeting minutes after the meeting.

[0037] First, the server collects users' calendar information and calculates common free time. Based on this information, it identifies mandatory participants according to their importance. For example, project leaders and key personnel will always be prioritized as meeting attendees.

[0038] Next, the server reserves an available meeting space during the acquired free time. This utilizes an interface with the company's internal meeting room reservation system. Details of the reserved meeting room are automatically added to the user's calendar.

[0039] During the meeting, the terminal uses speech recognition technology to transcribe speech in real time and sends the content to the server. The server monitors these speeches and has a function to notify users via the terminal if there is a deviation from the agenda. For example, if the content deviates from the scheduled agenda, it will warn the user with a message such as "You need to return to the agenda."

[0040] After the meeting ends, the server automatically generates meeting minutes using the text data collected during the meeting, summarizing key points as needed. These minutes and summaries are quickly distributed to relevant parties, facilitating information sharing.

[0041] Furthermore, users can provide feedback on their meeting experience through their devices. The server collects and analyzes this feedback to help improve future meetings.

[0042] Implementing this system can significantly reduce meeting preparation time and improve the efficiency of meeting management. For example, one company was able to halve the time spent on meeting-related tasks and improve employee productivity by introducing this system.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server retrieves users' calendar information and checks each user's schedule. Next, it compares these schedules and calculates common free time for everyone. Based on the importance of each user, it identifies those who must participate.

[0046] Step 2:

[0047] The server searches for available meeting locations based on the calculated free time. This is done through the meeting room reservation system. The server selects an available meeting room and automatically adds it to the user's calendar.

[0048] Step 3:

[0049] At the start of the meeting, the device activates its speech recognition function and transcribes the content spoken during the meeting into text in real time. The device then sends the recognized text to the server.

[0050] Step 4:

[0051] The server compares the received text against a predetermined agenda and checks the progress. If the discussion deviates from the agenda, the server notifies the user via the terminal that "the topic has deviated from the agenda."

[0052] Step 5:

[0053] The server monitors and manages the time allocated to each agenda item. When the scheduled time approaches, it sends an alert to the user via their terminal saying, "Please move on to the next agenda item."

[0054] Step 6:

[0055] After the meeting ends, the server automatically creates meeting minutes based on the text data generated during the meeting. These minutes and a summary are then sent to participants via email.

[0056] Step 7:

[0057] Users input feedback on the meeting via their devices. The server analyzes the collected feedback and considers ways to improve the next meeting.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] Meetings in companies and organizations are plagued by inefficiencies in preparation time and execution. It is necessary to reduce the time and effort involved in scheduling participants, securing meeting locations, managing the meeting, and creating and sharing minutes. Furthermore, it is essential to prevent meetings from deviating from the agenda and improve their quality. Additionally, an appropriate mechanism is needed to utilize participant feedback to improve future meetings.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for acquiring the activity schedules of multiple users and calculating shared free time; means for automatically reserving a meeting place based on the free time; and means for transcribing statements made during discussions using speech recognition and monitoring the progress. This enables automation of meeting preparation, highly accurate time management, efficient progress management, and rapid and accurate information sharing.

[0063] "User" refers to a person who uses this system to participate in a meeting.

[0064] "Planned activities" refers to a collection of schedule data that includes the date and time the user has planned to do so, as well as related event information.

[0065] "Shared free time" refers to common unscheduled time found by comparing the activity schedules of multiple users.

[0066] "Meeting place" refers to the location where the meeting is held, such as a conference room or an online meeting platform.

[0067] "Speech recognition" refers to the technology that converts spoken words during a discussion into text data in real time.

[0068] "Discussion" refers to an activity in which participants talk about an agenda item.

[0069] "Monitoring the progress" means that the server monitors the discussion in real time to ensure that it does not deviate from the set agenda and time allocation.

[0070] "Records" refer to documents and data compiled based on the content of discussions, and include summarized minutes as necessary.

[0071] A "generative artificial intelligence model" refers to an artificial intelligence system that uses generative AI models to perform summarization and information extraction based on input data.

[0072] "Key points" refer to information or content that should be given particular importance during a meeting or discussion.

[0073] In implementing this invention, the server collects user activity schedule data and calculates shared availability. Calendar services such as Google® Calendar API and Outlook Calendar API are utilized to obtain information from the user's terminal or calendar management application. Based on this information, the server optimizes the timing and automatically reserves the meeting place. General meeting room reservation management software is used in this process.

[0074] During the meeting, the terminal uses speech recognition technology to transcribe the user's speech in real time. This speech recognition process may utilize the Google Speech-to-Text API, among others. The generated text is sent to a server and monitored by a specific algorithm. This monitoring process incorporates a function to compare the text data with a pre-set agenda, and the user is notified if a deviation is detected.

[0075] After the meeting ends, the server automatically generates a record using the text data obtained during the meeting. By utilizing a generative artificial intelligence model, it is possible to summarize the meeting minutes and extract key points. One example of a model used here is OpenAI's GPT (Growth Proof Test).

[0076] Furthermore, users can provide feedback on the meeting through their devices. This feedback data is collected and analyzed by the server and used to improve future meetings.

[0077] As a concrete example, one company reported that after implementing this system, employees were able to reduce the time spent on meeting-related tasks and use the freed-up time to complete projects, resulting in increased productivity.

[0078] Examples of prompts used to manipulate a generating AI model include instructions such as, "Summarize the key points of the meeting." Such instructions enable the generated AI model to quickly and accurately produce meeting minutes.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The server collects user activity schedule data through the terminal. This process uses the Google Calendar API and Outlook Calendar API to obtain calendar information. The input is the user's calendar information, and the output is a list of available time slots based on each participant's activity schedule. Based on this list, the server calculates common available time slots and determines the optimal meeting time.

[0082] Step 2:

[0083] The server automatically reserves meeting places based on common availability. Here, it uses the company's internal meeting room reservation system to reserve a meeting room suitable for the available time slot. The input is information about availability and available meeting rooms, and the output is detailed information about the reserved meeting room. This information is automatically added to the user's calendar.

[0084] Step 3:

[0085] During a meeting, the device uses speech recognition technology to transcribe the user's speech in real time. The Google Speech-to-Text API is commonly used, where speech data is input and text data is output. The generated text is sent to a server, and the content of the speech is recorded in real time.

[0086] Step 4:

[0087] The server monitors the received text data and compares it against the agenda. The input is the text data generated during the meeting, and the output is the monitoring results. If a participant's comments deviate from the agenda, the server notifies them via their terminal with the message, "You need to return to the agenda." This feedback function ensures the efficiency of the meeting.

[0088] Step 5:

[0089] After the meeting ends, the server uses a generative AI model to create a record. The input is all the text data generated during the meeting, and the output is a meeting transcript including a summary. Using OpenAI's GPT model, key points are extracted and summarized, and delivered quickly to the user.

[0090] Step 6:

[0091] Users provide feedback on their meeting experience through their devices. Input consists of user feedback comments, and output is analysis results to help improve future meetings. The server collects and analyzes this data to generate suggestions for improving the overall meeting experience.

[0092] (Application Example 1)

[0093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] Online meetings and gatherings often present challenges such as the time-consuming process of coordinating participants' schedules and booking venues, as well as the difficulty of efficiently managing the meeting's progress and content. Furthermore, discussions can stray from the agenda, making progress management cumbersome. Additionally, creating meeting minutes afterward is time-consuming, often delaying information sharing among participants. Traditional systems have struggled to comprehensively address these issues.

[0095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0096] In this invention, the server includes means for obtaining the activity schedules of multiple users and calculating common free time; means for automatically reserving a meeting place based on the free time; means for transcribing speech during a meeting into text using speech recognition and monitoring the progress; means for collecting feedback from users on the meeting to help improve future meetings; and means for displaying the progress of the agenda in real time using a visual assistance device. This reduces meeting preparation time, allows for proper management of the meeting's progress, and enables efficient minute-taking and rapid information sharing with participants.

[0097] A "user" is an individual or organization that uses this system to participate in meetings or gatherings.

[0098] "Activity schedule" refers to the schedules and events that users are planning to participate in.

[0099] "Free time" refers to the period of time that is available in common when considering the activity schedules of multiple users.

[0100] A "meeting place" refers to the physical or virtual location or environment where a meeting or event takes place.

[0101] "Speech recognition" is a technology that allows machines to understand spoken content and convert it into text data.

[0102] "Visual assistance devices" refer to smart glasses and other devices that are used to visually supplement information.

[0103] "Progress monitoring" is the process of managing the progress of a meeting or gathering and ensuring that it proceeds according to the scheduled agenda.

[0104] "Meeting minutes" are documents that record the content of discussions and decisions made at meetings or gatherings.

[0105] "Real-time display" means that information is presented to the user almost instantly, with virtually no time delay.

[0106] This system uses a cloud-based server to manage the activity schedules of multiple users. The server collects the users' activity schedules and calculates common free time from this data. Based on the calculated free time, the server automatically reserves meeting places. For example, if a virtual meeting room is needed, it reserves a virtual meeting room through the existing interface.

[0107] Furthermore, while the meeting is in progress, speech recognition technology is used to convert audio data from the terminal into text in real time. The Google Speech-to-Text API is used for this speech recognition. The text data is analyzed on the server using natural language processing libraries (e.g., spaCy or NLTK) to monitor whether the progress is on track with the agenda. If the progress deviates from the agenda, the server notifies the user via a visual aid.

[0108] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the text data acquired in real time. These minutes are immediately distributed to all relevant parties. Users can submit feedback on the meeting through this system, and this feedback is collected by the server and used to improve future meetings.

[0109] For example, one company implemented this system, allowing employees to use smart glasses to check the progress of meetings in real time and receive meeting minutes instantly. This enabled employees to improve meeting efficiency and increase productivity.

[0110] An example of a prompt is: "At the next meeting, please demonstrate a system that transcribes audio into text in real time, monitors the agenda progress, and automatically shares meeting minutes after the meeting."

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The server retrieves users' activity schedules. It synchronizes with the calendar applications used by the users and imports each user's activity schedule data. The input is the user's calendar data, and the output is the basic data for calculating common free time. This process uses the calendar API to extract data.

[0114] Step 2:

[0115] The server analyzes the acquired activity schedule data and calculates common free time slots among multiple users. The input is the activity schedule data obtained in step 1, and the output is a list of time slots when all users are free. As part of the data processing, overlapping time slots are detected and common free time slots are listed.

[0116] Step 3:

[0117] The server automatically reserves meeting places based on common available time slots. The input is the available time data obtained in step 2, and the output is information about the reserved meeting places. As a data calculation, it selects meeting places that are available during the available time slots and performs the procedure using the reservation system API.

[0118] Step 4:

[0119] The device sends audio data from the meeting to a speech recognition service for real-time text conversion. The input is audio data acquired via the microphone, and the output is text data. This process involves analyzing the audio using a speech recognition API (e.g., Google Speech-to-Text).

[0120] Step 5:

[0121] The server analyzes the transcribed data and monitors the progress of the agenda. The input is the text data obtained in step 4, and the output is the analysis of the progress. Using a natural language processing library, the server compares the spoken content with the pre-set agenda to check for deviations from the agenda.

[0122] Step 6:

[0123] If a deviation from the progress is detected, the server will notify the user via a visual aid. The input is the progress analysis result obtained in step 5, and the output is a warning message. Specifically, a warning message is generated and displayed in real time on the user's visual aid.

[0124] Step 7:

[0125] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the collected text data and distributes them to users. The input is the text data obtained in step 4, and the output is the automatically generated meeting minutes. This process extracts important statements and decisions and uses email or a dedicated system to quickly distribute them to users.

[0126] Step 8:

[0127] Users submit feedback about meetings to the system. The input is user feedback information, and the output is feedback data stored on the server. This feedback is used to improve future meetings, so the server collects it and incorporates it into the next meeting plan.

[0128] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0129] This invention is a system aimed at improving the efficiency of meetings, and a key feature is the incorporation of an emotion engine. This system automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress, and creation and distribution of meeting minutes after the meeting, while also recognizing and analyzing participants' emotions to aid in the smooth running of the meeting.

[0130] First, the server retrieves users' calendar information and analyzes each user's schedule to identify common free time slots. Then, it automatically reserves a suitable meeting location for those free time slots and registers it in the user's calendar. The emotion engine activates as soon as the meeting begins, monitoring the emotional state of the participants.

[0131] The terminal uses speech recognition technology to transcribe speech in real time during the meeting and sends the content to the server. The server monitors the progress of the agenda and analyzes the emotional state of the participants. If a participant's emotions change in a negative direction, the server notifies the meeting facilitator through the terminal and adjusts the meeting's progress as needed.

[0132] During the meeting, the server manages the time allocated to each agenda item and optimizes the process based on sentiment analysis results. For example, if participants are feeling tired even though the agenda is progressing as planned, the server will send a notification suggesting a short break.

[0133] After the meeting, the server automatically creates meeting minutes based on the generated text data and summarizes the results of the emotional state analysis. These minutes and summaries are distributed to participants to help them adjust the strategy for the next meeting based on their feedback.

[0134] As a concrete example, by implementing this system in one company, they were able to improve customer satisfaction by maintaining participants' concentration during meetings and appropriately managing their emotional states. Thus, the present invention provides a comprehensive solution for improving the productivity of meetings.

[0135] The following describes the processing flow.

[0136] Step 1:

[0137] The server retrieves users' calendar information via an API and analyzes each user's schedule. This allows it to calculate common free time for all participants. It then identifies mandatory participants based on their importance.

[0138] Step 2:

[0139] The server checks the company's internal meeting room reservation system for available meeting spaces based on shared availability. It automatically reserves the selected meeting room and reflects that information in each user's calendar.

[0140] Step 3:

[0141] At the start of the meeting, the terminal activates its speech recognition function and transcribes all statements made during the meeting into text in real time. This text information is sent to a server and used for progress management.

[0142] Step 4:

[0143] The device uses its built-in emotion engine to recognize each user's emotions from their voice and speech recognition results. The identified emotion information is then sent to the server.

[0144] Step 5:

[0145] The server analyzes text and emotional information obtained from the audio and monitors the progress of the agenda. If the user's emotional state is negative, the server sends a notification to the facilitator via the terminal to draw their attention.

[0146] Step 6:

[0147] During the meeting, the server measures the time elapsed for each agenda item and sends an alert to users via their terminals if the scheduled time is likely to be exceeded. Based on the emotion engine data, it recommends adjusting the progress as needed.

[0148] Step 7:

[0149] After the meeting ends, the server automatically generates meeting minutes based on the speech recognition results. If there are any notable emotional shifts, a summary including those will also be created. The minutes will be distributed to all participants.

[0150] Step 8:

[0151] Users can provide feedback on meetings through their devices. The server uses the collected feedback and sentiment analysis results to formulate suggestions for improving the next meeting.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] In today's business environment, meetings play a crucial role in corporate operations, yet many are inefficient and contribute to decreased productivity. Improving the efficiency of meetings requires proper adjustment of many factors, including time management, location selection, progress monitoring, and participant emotional state management. However, traditional systems have struggled to comprehensively and automatically manage these factors.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes means for acquiring time management data from multiple information users and calculating common free time, means for automatically reserving a meeting place based on the free time, and means for documenting speech during the meeting using speech conversion technology and monitoring its progress. This improves the efficiency of the meeting and enables the operation of a highly productive meeting by managing the emotional state of the participants.

[0157] "Information users" refers to individual users who use the system and participate in the management or participation of a gathering.

[0158] "Time management data" refers to schedule information, including the user's appointments and free time.

[0159] "Shared free time" refers to time slots that are free at the same time on the schedules of multiple information users.

[0160] A "meeting place" refers to a physical or virtual location where a gathering takes place.

[0161] "Voice conversion technology" refers to the technology that converts spoken audio from a group into text data in real time.

[0162] "Means of monitoring progress" refers to a function that tracks the progress of a meeting in real time and makes adjustments as needed.

[0163] "Emotional analysis" refers to a technology that detects and analyzes the emotional state of participants.

[0164] "Meeting records" refer to documents that transcribe the statements and decisions made during a meeting into text.

[0165] "Means of gathering opinions" refers to the function of collecting feedback and suggestions from information users regarding the group.

[0166] This invention is a comprehensive system designed to enable information users to efficiently manage gatherings. Its main features include the high degree of automation of elements such as time management, location selection, progress monitoring, sentiment analysis, and opinion gathering.

[0167] The server utilizes various calendar APIs to obtain time management data from information users. These APIs collect information users' schedules and function as a database for calculating common free time. A specific example of such an API is the time management system API.

[0168] The server automatically reserves meeting places based on common availability. For example, it uses a reservation system API to check the availability of physical meeting rooms or online platforms and complete reservations.

[0169] The terminal uses speech conversion technology to document conversations in real time. Specific technologies that could be used include speech recognition software libraries and speech recognition APIs. This allows for rapid text conversion and transmission to a server.

[0170] The server monitors the progress of the meeting based on speech-recognized documents and uses an emotion analysis engine to analyze participants' emotional states in real time. This helps to provide appropriate advice to the meeting facilitator.

[0171] After the meeting concludes, the server distributes a meeting record generated by a generative AI model using speech recognition text to information users and collects feedback and opinions from participants. Natural language generation models can be used as an example of generative AI model usage.

[0172] In one specific case, implementing this system within a company resulted in a 30% reduction in typical meeting times while simultaneously lowering participant fatigue. This led to smoother workflow and improved labor productivity.

[0173] As an example of a prompt, the AI ​​model can be given the instruction, "Summarize the following meeting content and propose a strategy for the next meeting based on the opinions submitted by the participants."

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The server retrieves user time management data. The input is each user's schedule information. Specifically, it collects appointments from the user's calendar using a time management system API. The collected schedule information is analyzed to identify common free time periods for each user. The output is a list of common free time periods that could potentially form a group.

[0177] Step 2:

[0178] The server takes common available time slots as input and automatically reserves meeting places based on that availability. It checks for available meeting places at specific dates and times via the reservation system API and automatically confirms reservations based on availability. The output is a reservation completion notification, which is reflected in the user's calendar.

[0179] Step 3:

[0180] Once the meeting begins, the terminal uses speech recognition technology to document speech in real time. It receives audio data as input and converts it to text data using a speech recognition software library. The output is the text of the speech to be recorded.

[0181] Step 4:

[0182] The server receives text data of speech sent from terminals during the meeting and monitors its progress. It analyzes this data to track the progress of the agenda and the content of participants' comments. Furthermore, it uses a sentiment analysis engine to evaluate participants' emotional states in real time. If problems are detected, it notifies the meeting facilitator. The output includes a progress monitoring report and sentiment analysis results.

[0183] Step 5:

[0184] After the meeting concludes, the server automatically generates a meeting record using an AI model based on the accumulated spoken text data. The output is an editable meeting record, which is then distributed to all participants. The generated record is distributed in a format that includes prompts for collecting feedback from participants.

[0185] Step 6:

[0186] Users provide their opinions and feedback to the server after the meeting using the received meeting records. This feedback data is used as reference material for considering improvements in preparation for the next meeting. The output is a collection of materials for optimizing the planning of the next meeting.

[0187] (Application Example 2)

[0188] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0189] In meetings, accurately understanding participants' emotional states and making appropriate adjustments accordingly is difficult. This leads to decreased meeting efficiency and increased participant stress. Furthermore, traditional meeting management systems cannot provide real-time feedback based on participants' emotions, thus failing to improve the quality of meetings.

[0190] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the schedules of multiple users and calculating common free time, means for automatically reserving meeting locations based on free time, means for transcribing speech during a meeting into text using speech recognition and monitoring the progress, and means for analyzing the emotional state of users and adjusting the progress of the meeting based on the analysis results. This makes it possible to accurately grasp the emotions of participants during a meeting and optimize the progress based on that.

[0191] A "user" refers to an individual who utilizes the system, and is the entity that obtains schedules and participates in meetings.

[0192] "Schedule" refers to date and time information related to activities and events set by the user, and is used for scheduling meetings.

[0193] "Free time" refers to the period of time when no appointments are scheduled for any of the users, as determined by comparing their schedules.

[0194] The term "meeting location" is a concept that includes not only the physical place selected for holding a meeting, but also, in some cases, a virtual location.

[0195] "Speech recognition" is a technology that converts spoken words during a meeting into text data in real time.

[0196] "Facilitation" refers to management activities, including the scheduled agenda and time management of a meeting.

[0197] "Emotional state" is an indicator that shows the psychological and emotional condition of participants, and is used as a subject of analysis to revitalize and improve meetings.

[0198] "Analysis results" refer to the information and data obtained after analyzing emotional states, and serve as the basis for adjusting the meeting's progress.

[0199] "Analysis" refers to the act of analyzing a user's emotional state and understanding their condition and trends.

[0200] The system for carrying out this invention consists of a server, a user terminal, and associated software. The server first retrieves the schedules of multiple users, analyzes the data to identify common free time slots, and then automatically selects and reserves the optimal meeting location based on the free time. When a meeting begins, the user terminal uses speech recognition technology to transcribe the spoken words in the meeting in real time and sends the text to the server.

[0201] The server monitors the progress of the meeting based on the acquired text information and simultaneously analyzes the emotional state of the users. Using a generative AI model, it analyzes the participants' psychological state in real time and optimizes the meeting's progress based on the results. For example, if the content of a discussion deviates from the agenda or if a user's emotions are leaning towards the negative, the server sends a notification to the facilitator's terminal to prompt necessary adjustments.

[0202] Specifically, if the sentiment analysis indicates that a participant is feeling tired, feedback such as suggesting a short break can be provided. Furthermore, after the meeting, the server automatically creates meeting minutes based on the generated text data and distributes them to all participants. The minutes will also include the sentiment analysis feedback, which can be used to improve future meetings and forums.

[0203] The specific hardware and software used include schedule management using the Google Calendar API, speech-to-text conversion using Amazon Polly, and sentiment analysis using the OpenAI API.

[0204] As a concrete example, using this system in a community discussion meeting allows for real-time monitoring of each participant's opinions and emotional state, enabling efficient and positive progress. The facilitator can use smart devices to encourage active discussion and adjust the agenda in a timely manner.

[0205] Examples of prompt statements for a generative AI model are as follows:

[0206] "Analyze the emotions in the following conversation and identify possible emotional states: Meeting transcript text."

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The server retrieves each user's schedule via the Google Calendar API. The input is the user's calendar information, and the output is common free time data. The server then calculates and identifies the time slots when all users are available to participate.

[0210] Step 2:

[0211] The server automatically selects the optimal meeting location and makes a reservation based on availability data. The input is the specified availability time, and the output is reservation information. This information is registered as a calendar event on the user's device.

[0212] Step 3:

[0213] After the meeting begins, the user's device uses speech recognition technology (Amazon Polly) to transcribe the meeting's speech into text in real time. The input is the audio from the meeting, and the output is text data. This data is sent to a server and serves as basic information for monitoring the meeting's progress.

[0214] Step 4:

[0215] The server uses an API (OpenAI) for a generative AI model to analyze the sentiment of meeting participants based on the acquired text data. The input is text data, and the output is the sentiment analysis result. The server analyzes this result to determine how to adjust the meeting's progress.

[0216] Step 5:

[0217] Based on the sentiment analysis results, feedback is provided to the facilitator as needed. The input is the sentiment analysis results, and the output is a suggestion for adjusting the discussion. The terminal sends notifications to the facilitator, suggesting ways to promote the discussion or take a break.

[0218] Step 6:

[0219] After the meeting ends, the server automatically creates meeting minutes based on the generated text data. The input is the text data collected during the meeting, and the output is the meeting minutes. The meeting minutes, including sentiment analysis feedback, are delivered to the user.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] This invention is a system aimed at improving the efficiency of meetings, which automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress during meetings, and creation and distribution of meeting minutes after the meeting.

[0237] First, the server collects users' calendar information and calculates common free time. Based on this information, it identifies mandatory participants according to their importance. For example, project leaders and key personnel will always be prioritized as meeting attendees.

[0238] Next, the server reserves an available meeting space during the acquired free time. This utilizes an interface with the company's internal meeting room reservation system. Details of the reserved meeting room are automatically added to the user's calendar.

[0239] During the meeting, the terminal uses speech recognition technology to transcribe speech in real time and sends the content to the server. The server monitors these speeches and has a function to notify users via the terminal if there is a deviation from the agenda. For example, if the content deviates from the scheduled agenda, it will warn the user with a message such as "You need to return to the agenda."

[0240] After the meeting ends, the server automatically generates meeting minutes using the text data collected during the meeting, summarizing key points as needed. These minutes and summaries are quickly distributed to relevant parties, facilitating information sharing.

[0241] Furthermore, users can provide feedback on their meeting experience through their devices. The server collects and analyzes this feedback to help improve future meetings.

[0242] Implementing this system can significantly reduce meeting preparation time and improve the efficiency of meeting management. For example, one company was able to halve the time spent on meeting-related tasks and improve employee productivity by introducing this system.

[0243] The following describes the processing flow.

[0244] Step 1:

[0245] The server retrieves users' calendar information and checks each user's schedule. Next, it compares these schedules and calculates common free time for everyone. Based on the importance of each user, it identifies those who must participate.

[0246] Step 2:

[0247] The server searches for available meeting locations based on the calculated free time. This is done through the meeting room reservation system. The server selects an available meeting room and automatically adds it to the user's calendar.

[0248] Step 3:

[0249] At the start of the meeting, the device activates its speech recognition function and transcribes the content spoken during the meeting into text in real time. The device then sends the recognized text to the server.

[0250] Step 4:

[0251] The server compares the received text against a predetermined agenda and checks the progress. If the discussion deviates from the agenda, the server notifies the user via the terminal that "the topic has deviated from the agenda."

[0252] Step 5:

[0253] The server monitors and manages the time allocated to each agenda item. When the scheduled time approaches, it sends an alert to the user via their terminal saying, "Please move on to the next agenda item."

[0254] Step 6:

[0255] After the meeting ends, the server automatically creates meeting minutes based on the text data generated during the meeting. These minutes and a summary are then sent to participants via email.

[0256] Step 7:

[0257] Users input feedback on the meeting via their devices. The server analyzes the collected feedback and considers ways to improve the next meeting.

[0258] (Example 1)

[0259] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0260] Meetings in companies and organizations are plagued by inefficiencies in preparation time and execution. It is necessary to reduce the time and effort involved in scheduling participants, securing meeting locations, managing the meeting, and creating and sharing minutes. Furthermore, it is essential to prevent meetings from deviating from the agenda and improve their quality. Additionally, an appropriate mechanism is needed to utilize participant feedback to improve future meetings.

[0261] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0262] In this invention, the server includes means for acquiring the activity schedules of multiple users and calculating shared free time; means for automatically reserving a meeting place based on the free time; and means for transcribing statements made during discussions using speech recognition and monitoring the progress. This enables automation of meeting preparation, highly accurate time management, efficient progress management, and rapid and accurate information sharing.

[0263] "User" refers to a person who uses this system to participate in a meeting.

[0264] "Planned activities" refers to a collection of schedule data that includes the date and time the user has planned to do so, as well as related event information.

[0265] "Shared free time" refers to common unscheduled time found by comparing the activity schedules of multiple users.

[0266] "Meeting place" refers to the location where the meeting is held, such as a conference room or an online meeting platform.

[0267] "Speech recognition" refers to the technology that converts spoken words during a discussion into text data in real time.

[0268] "Discussion" refers to an activity in which participants talk about an agenda item.

[0269] "Monitoring the progress" means that the server monitors the discussion in real time to ensure that it does not deviate from the set agenda and time allocation.

[0270] "Records" refer to documents and data compiled based on the content of discussions, and include summarized minutes as necessary.

[0271] A "generative artificial intelligence model" refers to an artificial intelligence system that uses generative AI models to perform summarization and information extraction based on input data.

[0272] "Key points" refer to information or content that should be given particular importance during a meeting or discussion.

[0273] In implementing this invention, the server collects user activity schedule data and calculates shared availability. Calendar services such as the Google Calendar API and Outlook Calendar API are utilized to obtain information from the user's terminal or calendar management application. Based on this information, the server optimizes the timing and automatically reserves the meeting place. General meeting room reservation management software is used in this process.

[0274] During the meeting, the terminal uses speech recognition technology to transcribe the user's speech in real time. This speech recognition process may utilize the Google Speech-to-Text API, among others. The generated text is sent to a server and monitored by a specific algorithm. This monitoring process incorporates a function to compare the text data with a pre-set agenda, and the user is notified if a deviation is detected.

[0275] After the meeting ends, the server automatically generates a record using the text data obtained during the meeting. By utilizing a generative artificial intelligence model, it is possible to summarize the meeting minutes and extract key points. One example of a model used here is OpenAI's GPT.

[0276] Furthermore, users can provide feedback on the meeting through their devices. This feedback data is collected and analyzed by the server and used to improve future meetings.

[0277] As a concrete example, one company reported that after implementing this system, employees were able to reduce the time spent on meeting-related tasks and use the freed-up time to complete projects, resulting in increased productivity.

[0278] Examples of prompts used to manipulate a generating AI model include instructions such as, "Summarize the key points of the meeting." Such instructions enable the generated AI model to quickly and accurately produce meeting minutes.

[0279] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0280] Step 1:

[0281] The server collects user activity schedule data through the terminal. This process uses the Google Calendar API and Outlook Calendar API to obtain calendar information. The input is the user's calendar information, and the output is a list of available time slots based on each participant's activity schedule. Based on this list, the server calculates common available time slots and determines the optimal meeting time.

[0282] Step 2:

[0283] The server automatically reserves a meeting place based on the common free time. Here, the in-house meeting room reservation system is used to reserve a meeting room suitable for the free time. The input is the information of the free time and the available meeting rooms, and the output is the detailed information of the reserved meeting room. This information is automatically added to the user's calendar.

[0284] Step 3:

[0285] During the meeting, the terminal uses speech recognition technology to transcribe the user's speech in real time. It is common to use the Google Speech-to-Text API, where voice data is input and text data is output. The generated text is sent to the server, and the content of the speech is recorded in real time.

[0286] Step 4:

[0287] The server monitors the received text data and matches it with the agenda. The input is the text data generated during the meeting, and the output is the monitoring result. If the speech deviates from the agenda, the server notifies the participants through the terminal that "it is necessary to return to the agenda". This feedback function ensures the efficiency of the meeting progress.

[0288] Step 5:

[0289] After the meeting, the server uses a generated AI model to create a record. The input is all the text data generated during the meeting, and the output is the minutes including a summary. The GPT model of OpenAI is used to extract and summarize the important points and quickly distribute them to the users.

[0290] Step 6:

[0291] The user provides feedback on the meeting experience through the terminal. The input is the feedback comments from the user, and the output is the analysis result for use in improving the next meeting. The server collects and analyzes this data and generates proposals for improving the entire meeting.

[0292] (Application Example 1)

[0293] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0294] Online meetings and gatherings often present challenges such as the time-consuming process of coordinating participants' schedules and booking venues, as well as the difficulty of efficiently managing the meeting's progress and content. Furthermore, discussions can stray from the agenda, making progress management cumbersome. Additionally, creating meeting minutes afterward is time-consuming, often delaying information sharing among participants. Traditional systems have struggled to comprehensively address these issues.

[0295] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0296] In this invention, the server includes means for obtaining the activity schedules of multiple users and calculating common free time; means for automatically reserving a meeting place based on the free time; means for transcribing speech during a meeting into text using speech recognition and monitoring the progress; means for collecting feedback from users on the meeting to help improve future meetings; and means for displaying the progress of the agenda in real time using a visual assistance device. This reduces meeting preparation time, allows for proper management of the meeting's progress, and enables efficient minute-taking and rapid information sharing with participants.

[0297] A "user" is an individual or organization that uses this system to participate in meetings or gatherings.

[0298] "Activity schedule" refers to the schedules and events that users are planning to participate in.

[0299] "Free time" refers to the period of time that is available in common when considering the activity schedules of multiple users.

[0300] A "meeting place" refers to a physical or virtual location or environment where meetings and events are held.

[0301] "Speech recognition" is a technology for a machine to understand the content of speech and convert it into text data.

[0302] A "visual assistance device" refers to smart glasses and other devices, which are devices for visually complementing information.

[0303] "Progress monitoring" is a process for managing the progress of meetings and gatherings and advancing them along the scheduled topics.

[0304] A "minutes" is a document recording the content of speeches and decisions made in a meeting or gathering.

[0305] "Display in real time" means that information is presented to users almost instantaneously with little time lag.

[0306] This system uses a server operating on the cloud to manage the activity schedules of multiple users. The server collects the activity schedules of users and calculates common free time from these data. Based on the calculated free time, the server automatically reserves a meeting place. For example, when a virtual meeting room is needed, it secures a virtual meeting room through an existing interface.

[0307] Furthermore, when a meeting is in progress, it uses speech recognition technology to convert voice data from the terminal into text in real time. Google Speech-to-Text API is used for this speech recognition. The text data is analyzed on the server using a natural language processing library (such as spaCy or NLTK) to monitor whether the progress is along the topics. If the progress deviates from the topics, the server notifies the users of a warning through a visual assistance device.

[0308] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the text data acquired in real time. These minutes are immediately distributed to all relevant parties. Users can submit feedback on the meeting through this system, and this feedback is collected by the server and used to improve future meetings.

[0309] For example, one company implemented this system, allowing employees to use smart glasses to check the progress of meetings in real time and receive meeting minutes instantly. This enabled employees to improve meeting efficiency and increase productivity.

[0310] An example of a prompt is: "At the next meeting, please demonstrate a system that transcribes audio into text in real time, monitors the agenda progress, and automatically shares meeting minutes after the meeting."

[0311] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0312] Step 1:

[0313] The server retrieves users' activity schedules. It synchronizes with the calendar applications used by the users and imports each user's activity schedule data. The input is the user's calendar data, and the output is the basic data for calculating common free time. This process uses the calendar API to extract data.

[0314] Step 2:

[0315] The server analyzes the acquired activity schedule data and calculates common free time slots among multiple users. The input is the activity schedule data obtained in step 1, and the output is a list of time slots when all users are free. As part of the data processing, overlapping time slots are detected and common free time slots are listed.

[0316] Step 3:

[0317] The server automatically reserves meeting places based on common available time slots. The input is the available time data obtained in step 2, and the output is information about the reserved meeting places. As a data calculation, it selects meeting places that are available during the available time slots and performs the procedure using the reservation system API.

[0318] Step 4:

[0319] The device sends audio data from the meeting to a speech recognition service for real-time text conversion. The input is audio data acquired via the microphone, and the output is text data. This process involves analyzing the audio using a speech recognition API (e.g., Google Speech-to-Text).

[0320] Step 5:

[0321] The server analyzes the transcribed data and monitors the progress of the agenda. The input is the text data obtained in step 4, and the output is the analysis of the progress. Using a natural language processing library, the server compares the spoken content with the pre-set agenda to check for deviations from the agenda.

[0322] Step 6:

[0323] If a deviation from the progress is detected, the server will notify the user via a visual aid. The input is the progress analysis result obtained in step 5, and the output is a warning message. Specifically, a warning message is generated and displayed in real time on the user's visual aid.

[0324] Step 7:

[0325] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the collected text data and distributes them to users. The input is the text data obtained in step 4, and the output is the automatically generated meeting minutes. This process extracts important statements and decisions and uses email or a dedicated system to quickly distribute them to users.

[0326] Step 8:

[0327] Users submit feedback about meetings to the system. The input is user feedback information, and the output is feedback data stored on the server. This feedback is used to improve future meetings, so the server collects it and incorporates it into the next meeting plan.

[0328] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0329] This invention is a system aimed at improving the efficiency of meetings, and a key feature is the incorporation of an emotion engine. This system automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress, and creation and distribution of meeting minutes after the meeting, while also recognizing and analyzing participants' emotions to aid in the smooth running of the meeting.

[0330] First, the server retrieves users' calendar information and analyzes each user's schedule to identify common free time slots. Then, it automatically reserves a suitable meeting location for those free time slots and registers it in the user's calendar. The emotion engine activates as soon as the meeting begins, monitoring the emotional state of the participants.

[0331] The terminal uses speech recognition technology to transcribe speech in real time during the meeting and sends the content to the server. The server monitors the progress of the agenda and analyzes the emotional state of the participants. If a participant's emotions change in a negative direction, the server notifies the meeting facilitator through the terminal and adjusts the meeting's progress as needed.

[0332] During the meeting, the server manages the time allocated to each agenda item and optimizes the process based on sentiment analysis results. For example, if participants are feeling tired even though the agenda is progressing as planned, the server will send a notification suggesting a short break.

[0333] After the meeting, the server automatically creates meeting minutes based on the generated text data and summarizes the results of the emotional state analysis. These minutes and summaries are distributed to participants to help them adjust the strategy for the next meeting based on their feedback.

[0334] As a concrete example, by implementing this system in one company, they were able to improve customer satisfaction by maintaining participants' concentration during meetings and appropriately managing their emotional states. Thus, the present invention provides a comprehensive solution for improving the productivity of meetings.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] The server retrieves users' calendar information via an API and analyzes each user's schedule. This allows it to calculate common free time for all participants. It then identifies mandatory participants based on their importance.

[0338] Step 2:

[0339] The server checks the company's internal meeting room reservation system for available meeting spaces based on shared availability. It automatically reserves the selected meeting room and reflects that information in each user's calendar.

[0340] Step 3:

[0341] At the start of the meeting, the terminal activates its speech recognition function and transcribes all statements made during the meeting into text in real time. This text information is sent to a server and used for progress management.

[0342] Step 4:

[0343] The device uses its built-in emotion engine to recognize each user's emotions from their voice and speech recognition results. The identified emotion information is then sent to the server.

[0344] Step 5:

[0345] The server analyzes text and emotional information obtained from the audio and monitors the progress of the agenda. If the user's emotional state is negative, the server sends a notification to the facilitator via the terminal to draw their attention.

[0346] Step 6:

[0347] During the meeting, the server measures the time elapsed for each agenda item and sends an alert to users via their terminals if the scheduled time is likely to be exceeded. Based on the emotion engine data, it recommends adjusting the progress as needed.

[0348] Step 7:

[0349] After the meeting ends, the server automatically generates meeting minutes based on the speech recognition results. If there are any notable emotional shifts, a summary including those will also be created. The minutes will be distributed to all participants.

[0350] Step 8:

[0351] Users can provide feedback on meetings through their devices. The server uses the collected feedback and sentiment analysis results to formulate suggestions for improving the next meeting.

[0352] (Example 2)

[0353] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0354] In today's business environment, meetings play a crucial role in corporate operations, yet many are inefficient and contribute to decreased productivity. Improving the efficiency of meetings requires proper adjustment of many factors, including time management, location selection, progress monitoring, and participant emotional state management. However, traditional systems have struggled to comprehensively and automatically manage these factors.

[0355] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0356] In this invention, the server includes means for acquiring time management data from multiple information users and calculating common free time, means for automatically reserving a meeting place based on the free time, and means for documenting speech during the meeting using speech conversion technology and monitoring its progress. This improves the efficiency of the meeting and enables the operation of a highly productive meeting by managing the emotional state of the participants.

[0357] "Information users" refers to individual users who use the system and participate in the management or participation of a gathering.

[0358] "Time management data" refers to schedule information, including the user's appointments and free time.

[0359] "Shared free time" refers to time slots that are free at the same time on the schedules of multiple information users.

[0360] A "meeting place" refers to a physical or virtual location where a gathering takes place.

[0361] "Voice conversion technology" refers to the technology that converts spoken audio from a group into text data in real time.

[0362] "Means of monitoring progress" refers to a function that tracks the progress of a meeting in real time and makes adjustments as needed.

[0363] "Emotional analysis" refers to a technology that detects and analyzes the emotional state of participants.

[0364] "Meeting records" refer to documents that transcribe the statements and decisions made during a meeting into text.

[0365] "Means of gathering opinions" refers to the function of collecting feedback and suggestions from information users regarding the group.

[0366] This invention is a comprehensive system designed to enable information users to efficiently manage gatherings. Its main features include the high degree of automation of elements such as time management, location selection, progress monitoring, sentiment analysis, and opinion gathering.

[0367] The server utilizes various calendar APIs to obtain time management data from information users. These APIs collect information users' schedules and function as a database for calculating common free time. A specific example of such an API is the time management system API.

[0368] The server automatically reserves meeting places based on common availability. For example, it uses a reservation system API to check the availability of physical meeting rooms or online platforms and complete reservations.

[0369] The terminal uses speech conversion technology to document conversations in real time. Specific technologies that could be used include speech recognition software libraries and speech recognition APIs. This allows for rapid text conversion and transmission to a server.

[0370] The server monitors the progress of the meeting based on speech-recognized documents and uses an emotion analysis engine to analyze participants' emotional states in real time. This helps to provide appropriate advice to the meeting facilitator.

[0371] After the meeting concludes, the server distributes a meeting record generated by a generative AI model using speech recognition text to information users and collects feedback and opinions from participants. Natural language generation models can be used as an example of generative AI model usage.

[0372] In one specific case, implementing this system within a company resulted in a 30% reduction in typical meeting times while simultaneously lowering participant fatigue. This led to smoother workflow and improved labor productivity.

[0373] As an example of a prompt, the AI ​​model can be given the instruction, "Summarize the following meeting content and propose a strategy for the next meeting based on the opinions submitted by the participants."

[0374] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0375] Step 1:

[0376] The server retrieves user time management data. The input is each user's schedule information. Specifically, it collects appointments from the user's calendar using a time management system API. The collected schedule information is analyzed to identify common free time periods for each user. The output is a list of common free time periods that could potentially form a group.

[0377] Step 2:

[0378] The server takes common available time slots as input and automatically reserves meeting places based on that availability. It checks for available meeting places at specific dates and times via the reservation system API and automatically confirms reservations based on availability. The output is a reservation completion notification, which is reflected in the user's calendar.

[0379] Step 3:

[0380] Once the meeting begins, the terminal uses speech recognition technology to document speech in real time. It receives audio data as input and converts it to text data using a speech recognition software library. The output is the text of the speech to be recorded.

[0381] Step 4:

[0382] The server receives text data of speech sent from terminals during the meeting and monitors its progress. It analyzes this data to track the progress of the agenda and the content of participants' comments. Furthermore, it uses a sentiment analysis engine to evaluate participants' emotional states in real time. If problems are detected, it notifies the meeting facilitator. The output includes a progress monitoring report and sentiment analysis results.

[0383] Step 5:

[0384] After the meeting concludes, the server automatically generates a meeting record using an AI model based on the accumulated spoken text data. The output is an editable meeting record, which is then distributed to all participants. The generated record is distributed in a format that includes prompts for collecting feedback from participants.

[0385] Step 6:

[0386] Users provide their opinions and feedback to the server after the meeting using the received meeting records. This feedback data is used as reference material for considering improvements in preparation for the next meeting. The output is a collection of materials for optimizing the planning of the next meeting.

[0387] (Application Example 2)

[0388] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0389] In meetings, accurately understanding participants' emotional states and making appropriate adjustments accordingly is difficult. This leads to decreased meeting efficiency and increased participant stress. Furthermore, traditional meeting management systems cannot provide real-time feedback based on participants' emotions, thus failing to improve the quality of meetings.

[0390] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the schedules of multiple users and calculating common free time, means for automatically reserving meeting locations based on free time, means for transcribing speech during a meeting into text using speech recognition and monitoring the progress, and means for analyzing the emotional state of users and adjusting the progress of the meeting based on the analysis results. This makes it possible to accurately grasp the emotions of participants during a meeting and optimize the progress based on that.

[0391] A "user" refers to an individual who utilizes the system, and is the entity that obtains schedules and participates in meetings.

[0392] "Schedule" refers to date and time information related to activities and events set by the user, and is used for scheduling meetings.

[0393] "Free time" refers to the period of time when no appointments are scheduled for any of the users, as determined by comparing their schedules.

[0394] The term "meeting location" is a concept that includes not only the physical place selected for holding a meeting, but also, in some cases, a virtual location.

[0395] "Speech recognition" is a technology that converts spoken words during a meeting into text data in real time.

[0396] "Facilitation" refers to management activities, including the scheduled agenda and time management of a meeting.

[0397] "Emotional state" is an indicator that shows the psychological and emotional condition of participants, and is used as a subject of analysis to revitalize and improve meetings.

[0398] "Analysis results" refer to the information and data obtained after analyzing emotional states, and serve as the basis for adjusting the meeting's progress.

[0399] "Analysis" refers to the act of analyzing a user's emotional state and understanding their condition and trends.

[0400] The system for carrying out this invention consists of a server, a user terminal, and associated software. The server first retrieves the schedules of multiple users, analyzes the data to identify common free time slots, and then automatically selects and reserves the optimal meeting location based on the free time. When a meeting begins, the user terminal uses speech recognition technology to transcribe the spoken words in the meeting in real time and sends the text to the server.

[0401] The server monitors the progress of the meeting based on the acquired text information and simultaneously analyzes the emotional state of the users. Using a generative AI model, it analyzes the participants' psychological state in real time and optimizes the meeting's progress based on the results. For example, if the content of a discussion deviates from the agenda or if a user's emotions are leaning towards the negative, the server sends a notification to the facilitator's terminal to prompt necessary adjustments.

[0402] Specifically, if the sentiment analysis indicates that a participant is feeling tired, feedback such as suggesting a short break can be provided. Furthermore, after the meeting, the server automatically creates meeting minutes based on the generated text data and distributes them to all participants. The minutes will also include the sentiment analysis feedback, which can be used to improve future meetings and forums.

[0403] The specific hardware and software used include schedule management using the Google Calendar API, speech-to-text conversion using Amazon Polly, and sentiment analysis using the OpenAI API.

[0404] As a concrete example, using this system in a community discussion meeting allows for real-time monitoring of each participant's opinions and emotional state, enabling efficient and positive progress. The facilitator can use smart devices to encourage active discussion and adjust the agenda in a timely manner.

[0405] Examples of prompt statements for a generative AI model are as follows:

[0406] "Analyze the emotions in the following conversation and identify possible emotional states: Meeting transcript text."

[0407] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0408] Step 1:

[0409] The server retrieves each user's schedule via the Google Calendar API. The input is the user's calendar information, and the output is common free time data. The server then calculates and identifies the time slots when all users are available to participate.

[0410] Step 2:

[0411] The server automatically selects the optimal meeting location and makes a reservation based on availability data. The input is the specified availability time, and the output is reservation information. This information is registered as a calendar event on the user's device.

[0412] Step 3:

[0413] After the meeting begins, the user's device uses speech recognition technology (Amazon Polly) to transcribe the meeting's speech into text in real time. The input is the audio from the meeting, and the output is text data. This data is sent to a server and serves as basic information for monitoring the meeting's progress.

[0414] Step 4:

[0415] The server uses an API (OpenAI) for a generative AI model to analyze the sentiment of meeting participants based on the acquired text data. The input is text data, and the output is the sentiment analysis result. The server analyzes this result to determine how to adjust the meeting's progress.

[0416] Step 5:

[0417] Based on the sentiment analysis results, feedback is provided to the facilitator as needed. The input is the sentiment analysis results, and the output is a suggestion for adjusting the discussion. The terminal sends notifications to the facilitator, suggesting ways to promote the discussion or take a break.

[0418] Step 6:

[0419] After the meeting ends, the server automatically creates meeting minutes based on the generated text data. The input is the text data collected during the meeting, and the output is the meeting minutes. The meeting minutes, including sentiment analysis feedback, are delivered to the user.

[0420] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0421] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0422] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0423] [Third Embodiment]

[0424] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0425] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0426] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0427] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0428] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0430] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0431] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0432] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0433] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0434] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0435] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0436] This invention is a system aimed at improving the efficiency of meetings, which automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress during meetings, and creation and distribution of meeting minutes after the meeting.

[0437] First, the server collects users' calendar information and calculates common free time. Based on this information, it identifies mandatory participants according to their importance. For example, project leaders and key personnel will always be prioritized as meeting attendees.

[0438] Next, the server reserves an available meeting space during the acquired free time. This utilizes an interface with the company's internal meeting room reservation system. Details of the reserved meeting room are automatically added to the user's calendar.

[0439] During the meeting, the terminal uses speech recognition technology to transcribe speech in real time and sends the content to the server. The server monitors these speeches and has a function to notify users via the terminal if there is a deviation from the agenda. For example, if the content deviates from the scheduled agenda, it will warn the user with a message such as "You need to return to the agenda."

[0440] After the meeting ends, the server automatically generates meeting minutes using the text data collected during the meeting, summarizing key points as needed. These minutes and summaries are quickly distributed to relevant parties, facilitating information sharing.

[0441] Furthermore, users can provide feedback on their meeting experience through their devices. The server collects and analyzes this feedback to help improve future meetings.

[0442] Implementing this system can significantly reduce meeting preparation time and improve the efficiency of meeting management. For example, one company was able to halve the time spent on meeting-related tasks and improve employee productivity by introducing this system.

[0443] The following describes the processing flow.

[0444] Step 1:

[0445] The server retrieves users' calendar information and checks each user's schedule. Next, it compares these schedules and calculates common free time for everyone. Based on the importance of each user, it identifies those who must participate.

[0446] Step 2:

[0447] The server searches for available meeting locations based on the calculated free time. This is done through the meeting room reservation system. The server selects an available meeting room and automatically adds it to the user's calendar.

[0448] Step 3:

[0449] At the start of the meeting, the device activates its speech recognition function and transcribes the content spoken during the meeting into text in real time. The device then sends the recognized text to the server.

[0450] Step 4:

[0451] The server compares the received text against a predetermined agenda and checks the progress. If the discussion deviates from the agenda, the server notifies the user via the terminal that "the topic has deviated from the agenda."

[0452] Step 5:

[0453] The server monitors and manages the time allocated to each agenda item. When the scheduled time approaches, it sends an alert to the user via their terminal saying, "Please move on to the next agenda item."

[0454] Step 6:

[0455] After the meeting ends, the server automatically creates meeting minutes based on the text data generated during the meeting. These minutes and a summary are then sent to participants via email.

[0456] Step 7:

[0457] Users input feedback on the meeting via their devices. The server analyzes the collected feedback and considers ways to improve the next meeting.

[0458] (Example 1)

[0459] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0460] Meetings in companies and organizations are plagued by inefficiencies in preparation time and execution. It is necessary to reduce the time and effort involved in scheduling participants, securing meeting locations, managing the meeting, and creating and sharing minutes. Furthermore, it is essential to prevent meetings from deviating from the agenda and improve their quality. Additionally, an appropriate mechanism is needed to utilize participant feedback to improve future meetings.

[0461] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0462] In this invention, the server includes means for acquiring the activity schedules of multiple users and calculating shared free time; means for automatically reserving a meeting place based on the free time; and means for transcribing statements made during discussions using speech recognition and monitoring the progress. This enables automation of meeting preparation, highly accurate time management, efficient progress management, and rapid and accurate information sharing.

[0463] "User" refers to a person who uses this system to participate in a meeting.

[0464] "Planned activities" refers to a collection of schedule data that includes the date and time the user has planned to do so, as well as related event information.

[0465] "Shared free time" refers to common unscheduled time found by comparing the activity schedules of multiple users.

[0466] "Meeting place" refers to the location where the meeting is held, such as a conference room or an online meeting platform.

[0467] "Speech recognition" refers to the technology that converts spoken words during a discussion into text data in real time.

[0468] "Discussion" refers to an activity in which participants talk about an agenda item.

[0469] "Monitoring the progress" means that the server monitors the discussion in real time to ensure that it does not deviate from the set agenda and time allocation.

[0470] "Records" refer to documents and data compiled based on the content of discussions, and include summarized minutes as necessary.

[0471] A "generative artificial intelligence model" refers to an artificial intelligence system that uses generative AI models to perform summarization and information extraction based on input data.

[0472] "Key points" refer to information or content that should be given particular importance during a meeting or discussion.

[0473] In implementing this invention, the server collects user activity schedule data and calculates shared availability. Calendar services such as the Google Calendar API and Outlook Calendar API are utilized to obtain information from the user's terminal or calendar management application. Based on this information, the server optimizes the timing and automatically reserves the meeting place. General meeting room reservation management software is used in this process.

[0474] During the meeting, the terminal uses speech recognition technology to transcribe the user's speech in real time. This speech recognition process may utilize the Google Speech-to-Text API, among others. The generated text is sent to a server and monitored by a specific algorithm. This monitoring process incorporates a function to compare the text data with a pre-set agenda, and the user is notified if a deviation is detected.

[0475] After the meeting ends, the server automatically generates a record using the text data obtained during the meeting. By utilizing a generative artificial intelligence model, it is possible to summarize the meeting minutes and extract key points. One example of a model used here is OpenAI's GPT.

[0476] Furthermore, users can provide feedback on the meeting through their devices. This feedback data is collected and analyzed by the server and used to improve future meetings.

[0477] As a concrete example, one company reported that after implementing this system, employees were able to reduce the time spent on meeting-related tasks and use the freed-up time to complete projects, resulting in increased productivity.

[0478] Examples of prompts used to manipulate a generating AI model include instructions such as, "Summarize the key points of the meeting." Such instructions enable the generated AI model to quickly and accurately produce meeting minutes.

[0479] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0480] Step 1:

[0481] The server collects user activity schedule data through the terminal. This process uses the Google Calendar API and Outlook Calendar API to obtain calendar information. The input is the user's calendar information, and the output is a list of available time slots based on each participant's activity schedule. Based on this list, the server calculates common available time slots and determines the optimal meeting time.

[0482] Step 2:

[0483] The server automatically reserves meeting places based on common availability. Here, it uses the company's internal meeting room reservation system to reserve a meeting room suitable for the available time slot. The input is information about availability and available meeting rooms, and the output is detailed information about the reserved meeting room. This information is automatically added to the user's calendar.

[0484] Step 3:

[0485] During a meeting, the device uses speech recognition technology to transcribe the user's speech in real time. The Google Speech-to-Text API is commonly used, where speech data is input and text data is output. The generated text is sent to a server, and the content of the speech is recorded in real time.

[0486] Step 4:

[0487] The server monitors the received text data and compares it against the agenda. The input is the text data generated during the meeting, and the output is the monitoring results. If a participant's comments deviate from the agenda, the server notifies them via their terminal with the message, "You need to return to the agenda." This feedback function ensures the efficiency of the meeting.

[0488] Step 5:

[0489] After the meeting ends, the server uses a generative AI model to create a record. The input is all the text data generated during the meeting, and the output is a meeting transcript including a summary. Using OpenAI's GPT model, key points are extracted and summarized, and delivered quickly to the user.

[0490] Step 6:

[0491] Users provide feedback on their meeting experience through their devices. Input consists of user feedback comments, and output is analysis results to help improve future meetings. The server collects and analyzes this data to generate suggestions for improving the overall meeting experience.

[0492] (Application Example 1)

[0493] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0494] Online meetings and gatherings often present challenges such as the time-consuming process of coordinating participants' schedules and booking venues, as well as the difficulty of efficiently managing the meeting's progress and content. Furthermore, discussions can stray from the agenda, making progress management cumbersome. Additionally, creating meeting minutes afterward is time-consuming, often delaying information sharing among participants. Traditional systems have struggled to comprehensively address these issues.

[0495] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0496] In this invention, the server includes means for obtaining the activity schedules of multiple users and calculating common free time; means for automatically reserving a meeting place based on the free time; means for transcribing speech during a meeting into text using speech recognition and monitoring the progress; means for collecting feedback from users on the meeting to help improve future meetings; and means for displaying the progress of the agenda in real time using a visual assistance device. This reduces meeting preparation time, allows for proper management of the meeting's progress, and enables efficient minute-taking and rapid information sharing with participants.

[0497] A "user" is an individual or organization that uses this system to participate in meetings or gatherings.

[0498] "Activity schedule" refers to the schedules and events that users are planning to participate in.

[0499] "Free time" refers to the period of time that is available in common when considering the activity schedules of multiple users.

[0500] A "meeting place" refers to the physical or virtual location or environment where a meeting or event takes place.

[0501] "Speech recognition" is a technology that allows machines to understand spoken content and convert it into text data.

[0502] "Visual assistance devices" refer to smart glasses and other devices that are used to visually supplement information.

[0503] "Progress monitoring" is the process of managing the progress of a meeting or gathering and ensuring that it proceeds according to the scheduled agenda.

[0504] "Meeting minutes" are documents that record the content of discussions and decisions made at meetings or gatherings.

[0505] "Real-time display" means that information is presented to the user almost instantly, with virtually no time delay.

[0506] This system uses a cloud-based server to manage the activity schedules of multiple users. The server collects the users' activity schedules and calculates common free time from this data. Based on the calculated free time, the server automatically reserves meeting places. For example, if a virtual meeting room is needed, it reserves a virtual meeting room through the existing interface.

[0507] Furthermore, while the meeting is in progress, speech recognition technology is used to convert audio data from the terminal into text in real time. The Google Speech-to-Text API is used for this speech recognition. The text data is analyzed on the server using natural language processing libraries (e.g., spaCy or NLTK) to monitor whether the progress is on track with the agenda. If the progress deviates from the agenda, the server notifies the user via a visual aid.

[0508] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the text data acquired in real time. These minutes are immediately distributed to all relevant parties. Users can submit feedback on the meeting through this system, and this feedback is collected by the server and used to improve future meetings.

[0509] For example, one company implemented this system, allowing employees to use smart glasses to check the progress of meetings in real time and receive meeting minutes instantly. This enabled employees to improve meeting efficiency and increase productivity.

[0510] An example of a prompt is: "At the next meeting, please demonstrate a system that transcribes audio into text in real time, monitors the agenda progress, and automatically shares meeting minutes after the meeting."

[0511] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0512] Step 1:

[0513] The server retrieves users' activity schedules. It synchronizes with the calendar applications used by the users and imports each user's activity schedule data. The input is the user's calendar data, and the output is the basic data for calculating common free time. This process uses the calendar API to extract data.

[0514] Step 2:

[0515] The server analyzes the acquired activity schedule data and calculates common free time slots among multiple users. The input is the activity schedule data obtained in step 1, and the output is a list of time slots when all users are free. As part of the data processing, overlapping time slots are detected and common free time slots are listed.

[0516] Step 3:

[0517] The server automatically reserves meeting places based on common available time slots. The input is the available time data obtained in step 2, and the output is information about the reserved meeting places. As a data calculation, it selects meeting places that are available during the available time slots and performs the procedure using the reservation system API.

[0518] Step 4:

[0519] The device sends audio data from the meeting to a speech recognition service for real-time text conversion. The input is audio data acquired via the microphone, and the output is text data. This process involves analyzing the audio using a speech recognition API (e.g., Google Speech-to-Text).

[0520] Step 5:

[0521] The server analyzes the transcribed data and monitors the progress of the agenda. The input is the text data obtained in step 4, and the output is the analysis of the progress. Using a natural language processing library, the server compares the spoken content with the pre-set agenda to check for deviations from the agenda.

[0522] Step 6:

[0523] If a deviation from the progress is detected, the server will notify the user via a visual aid. The input is the progress analysis result obtained in step 5, and the output is a warning message. Specifically, a warning message is generated and displayed in real time on the user's visual aid.

[0524] Step 7:

[0525] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the collected text data and distributes them to users. The input is the text data obtained in step 4, and the output is the automatically generated meeting minutes. This process extracts important statements and decisions and uses email or a dedicated system to quickly distribute them to users.

[0526] Step 8:

[0527] Users submit feedback about meetings to the system. The input is user feedback information, and the output is feedback data stored on the server. This feedback is used to improve future meetings, so the server collects it and incorporates it into the next meeting plan.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] This invention is a system aimed at improving the efficiency of meetings, and a key feature is the incorporation of an emotion engine. This system automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress, and creation and distribution of meeting minutes after the meeting, while also recognizing and analyzing participants' emotions to aid in the smooth running of the meeting.

[0530] First, the server retrieves users' calendar information and analyzes each user's schedule to identify common free time slots. Then, it automatically reserves a suitable meeting location for those free time slots and registers it in the user's calendar. The emotion engine activates as soon as the meeting begins, monitoring the emotional state of the participants.

[0531] The terminal uses speech recognition technology to transcribe speech in real time during the meeting and sends the content to the server. The server monitors the progress of the agenda and analyzes the emotional state of the participants. If a participant's emotions change in a negative direction, the server notifies the meeting facilitator through the terminal and adjusts the meeting's progress as needed.

[0532] During the meeting, the server manages the time allocated to each agenda item and optimizes the process based on sentiment analysis results. For example, if participants are feeling tired even though the agenda is progressing as planned, the server will send a notification suggesting a short break.

[0533] After the meeting, the server automatically creates meeting minutes based on the generated text data and summarizes the results of the emotional state analysis. These minutes and summaries are distributed to participants to help them adjust the strategy for the next meeting based on their feedback.

[0534] As a concrete example, by implementing this system in one company, they were able to improve customer satisfaction by maintaining participants' concentration during meetings and appropriately managing their emotional states. Thus, the present invention provides a comprehensive solution for improving the productivity of meetings.

[0535] The following describes the processing flow.

[0536] Step 1:

[0537] The server retrieves users' calendar information via an API and analyzes each user's schedule. This allows it to calculate common free time for all participants. It then identifies mandatory participants based on their importance.

[0538] Step 2:

[0539] The server checks the company's internal meeting room reservation system for available meeting spaces based on shared availability. It automatically reserves the selected meeting room and reflects that information in each user's calendar.

[0540] Step 3:

[0541] At the start of the meeting, the terminal activates its speech recognition function and transcribes all statements made during the meeting into text in real time. This text information is sent to a server and used for progress management.

[0542] Step 4:

[0543] The device uses its built-in emotion engine to recognize each user's emotions from their voice and speech recognition results. The identified emotion information is then sent to the server.

[0544] Step 5:

[0545] The server analyzes text and emotional information obtained from the audio and monitors the progress of the agenda. If the user's emotional state is negative, the server sends a notification to the facilitator via the terminal to draw their attention.

[0546] Step 6:

[0547] During the meeting, the server measures the time elapsed for each agenda item and sends an alert to users via their terminals if the scheduled time is likely to be exceeded. Based on the emotion engine data, it recommends adjusting the progress as needed.

[0548] Step 7:

[0549] After the meeting ends, the server automatically generates meeting minutes based on the speech recognition results. If there are any notable emotional shifts, a summary including those will also be created. The minutes will be distributed to all participants.

[0550] Step 8:

[0551] Users can provide feedback on meetings through their devices. The server uses the collected feedback and sentiment analysis results to formulate suggestions for improving the next meeting.

[0552] (Example 2)

[0553] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0554] In today's business environment, meetings play a crucial role in corporate operations, yet many are inefficient and contribute to decreased productivity. Improving the efficiency of meetings requires proper adjustment of many factors, including time management, location selection, progress monitoring, and participant emotional state management. However, traditional systems have struggled to comprehensively and automatically manage these factors.

[0555] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0556] In this invention, the server includes means for acquiring time management data from multiple information users and calculating common free time, means for automatically reserving a meeting place based on the free time, and means for documenting speech during the meeting using speech conversion technology and monitoring its progress. This improves the efficiency of the meeting and enables the operation of a highly productive meeting by managing the emotional state of the participants.

[0557] "Information users" refers to individual users who use the system and participate in the management or participation of a gathering.

[0558] "Time management data" refers to schedule information, including the user's appointments and free time.

[0559] "Shared free time" refers to time slots that are free at the same time on the schedules of multiple information users.

[0560] A "meeting place" refers to a physical or virtual location where a gathering takes place.

[0561] "Voice conversion technology" refers to the technology that converts spoken audio from a group into text data in real time.

[0562] "Means of monitoring progress" refers to a function that tracks the progress of a meeting in real time and makes adjustments as needed.

[0563] "Emotional analysis" refers to a technology that detects and analyzes the emotional state of participants.

[0564] "Meeting records" refer to documents that transcribe the statements and decisions made during a meeting into text.

[0565] "Means of gathering opinions" refers to the function of collecting feedback and suggestions from information users regarding the group.

[0566] This invention is a comprehensive system designed to enable information users to efficiently manage gatherings. Its main features include the high degree of automation of elements such as time management, location selection, progress monitoring, sentiment analysis, and opinion gathering.

[0567] The server utilizes various calendar APIs to obtain time management data from information users. These APIs collect information users' schedules and function as a database for calculating common free time. A specific example of such an API is the time management system API.

[0568] The server automatically reserves meeting places based on common availability. For example, it uses a reservation system API to check the availability of physical meeting rooms or online platforms and complete reservations.

[0569] The terminal uses speech conversion technology to document conversations in real time. Specific technologies that could be used include speech recognition software libraries and speech recognition APIs. This allows for rapid text conversion and transmission to a server.

[0570] The server monitors the progress of the meeting based on speech-recognized documents and uses an emotion analysis engine to analyze participants' emotional states in real time. This helps to provide appropriate advice to the meeting facilitator.

[0571] After the meeting concludes, the server distributes a meeting record generated by a generative AI model using speech recognition text to information users and collects feedback and opinions from participants. Natural language generation models can be used as an example of generative AI model usage.

[0572] In one specific case, implementing this system within a company resulted in a 30% reduction in typical meeting times while simultaneously lowering participant fatigue. This led to smoother workflow and improved labor productivity.

[0573] As an example of a prompt, the AI ​​model can be given the instruction, "Summarize the following meeting content and propose a strategy for the next meeting based on the opinions submitted by the participants."

[0574] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0575] Step 1:

[0576] The server retrieves user time management data. The input is each user's schedule information. Specifically, it collects appointments from the user's calendar using a time management system API. The collected schedule information is analyzed to identify common free time periods for each user. The output is a list of common free time periods that could potentially form a group.

[0577] Step 2:

[0578] The server takes common available time slots as input and automatically reserves meeting places based on that availability. It checks for available meeting places at specific dates and times via the reservation system API and automatically confirms reservations based on availability. The output is a reservation completion notification, which is reflected in the user's calendar.

[0579] Step 3:

[0580] Once the meeting begins, the terminal uses speech recognition technology to document speech in real time. It receives audio data as input and converts it to text data using a speech recognition software library. The output is the text of the speech to be recorded.

[0581] Step 4:

[0582] The server receives text data of speech sent from terminals during the meeting and monitors its progress. It analyzes this data to track the progress of the agenda and the content of participants' comments. Furthermore, it uses a sentiment analysis engine to evaluate participants' emotional states in real time. If problems are detected, it notifies the meeting facilitator. The output includes a progress monitoring report and sentiment analysis results.

[0583] Step 5:

[0584] After the meeting concludes, the server automatically generates a meeting record using an AI model based on the accumulated spoken text data. The output is an editable meeting record, which is then distributed to all participants. The generated record is distributed in a format that includes prompts for collecting feedback from participants.

[0585] Step 6:

[0586] Users provide their opinions and feedback to the server after the meeting using the received meeting records. This feedback data is used as reference material for considering improvements in preparation for the next meeting. The output is a collection of materials for optimizing the planning of the next meeting.

[0587] (Application Example 2)

[0588] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] In meetings, accurately understanding participants' emotional states and making appropriate adjustments accordingly is difficult. This leads to decreased meeting efficiency and increased participant stress. Furthermore, traditional meeting management systems cannot provide real-time feedback based on participants' emotions, thus failing to improve the quality of meetings.

[0590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the schedules of multiple users and calculating common free time, means for automatically reserving meeting locations based on free time, means for transcribing speech during a meeting into text using speech recognition and monitoring the progress, and means for analyzing the emotional state of users and adjusting the progress of the meeting based on the analysis results. This makes it possible to accurately grasp the emotions of participants during a meeting and optimize the progress based on that.

[0591] A "user" refers to an individual who utilizes the system, and is the entity that obtains schedules and participates in meetings.

[0592] "Schedule" refers to date and time information related to activities and events set by the user, and is used for scheduling meetings.

[0593] "Free time" refers to the period of time when no appointments are scheduled for any of the users, as determined by comparing their schedules.

[0594] The term "meeting location" is a concept that includes not only the physical place selected for holding a meeting, but also, in some cases, a virtual location.

[0595] "Speech recognition" is a technology that converts spoken words during a meeting into text data in real time.

[0596] "Facilitation" refers to management activities, including the scheduled agenda and time management of a meeting.

[0597] "Emotional state" is an indicator that shows the psychological and emotional condition of participants, and is used as a subject of analysis to revitalize and improve meetings.

[0598] "Analysis results" refer to the information and data obtained after analyzing emotional states, and serve as the basis for adjusting the meeting's progress.

[0599] "Analysis" refers to the act of analyzing a user's emotional state and understanding their condition and trends.

[0600] The system for carrying out this invention consists of a server, a user terminal, and associated software. The server first retrieves the schedules of multiple users, analyzes the data to identify common free time slots, and then automatically selects and reserves the optimal meeting location based on the free time. When a meeting begins, the user terminal uses speech recognition technology to transcribe the spoken words in the meeting in real time and sends the text to the server.

[0601] The server monitors the progress of the meeting based on the acquired text information and simultaneously analyzes the emotional state of the users. Using a generative AI model, it analyzes the participants' psychological state in real time and optimizes the meeting's progress based on the results. For example, if the content of a discussion deviates from the agenda or if a user's emotions are leaning towards the negative, the server sends a notification to the facilitator's terminal to prompt necessary adjustments.

[0602] Specifically, if the sentiment analysis indicates that a participant is feeling tired, feedback such as suggesting a short break can be provided. Furthermore, after the meeting, the server automatically creates meeting minutes based on the generated text data and distributes them to all participants. The minutes will also include the sentiment analysis feedback, which can be used to improve future meetings and forums.

[0603] The specific hardware and software used include schedule management using the Google Calendar API, speech-to-text conversion using Amazon Polly, and sentiment analysis using the OpenAI API.

[0604] As a concrete example, using this system in a community discussion meeting allows for real-time monitoring of each participant's opinions and emotional state, enabling efficient and positive progress. The facilitator can use smart devices to encourage active discussion and adjust the agenda in a timely manner.

[0605] Examples of prompt statements for a generative AI model are as follows:

[0606] "Analyze the emotions in the following conversation and identify possible emotional states: Meeting transcript text."

[0607] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0608] Step 1:

[0609] The server retrieves each user's schedule via the Google Calendar API. The input is the user's calendar information, and the output is common free time data. The server then calculates and identifies the time slots when all users are available to participate.

[0610] Step 2:

[0611] The server automatically selects the optimal meeting location and makes a reservation based on availability data. The input is the specified availability time, and the output is reservation information. This information is registered as a calendar event on the user's device.

[0612] Step 3:

[0613] After the meeting begins, the user's device uses speech recognition technology (Amazon Polly) to transcribe the meeting's speech into text in real time. The input is the audio from the meeting, and the output is text data. This data is sent to a server and serves as basic information for monitoring the meeting's progress.

[0614] Step 4:

[0615] The server uses an API (OpenAI) for a generative AI model to analyze the sentiment of meeting participants based on the acquired text data. The input is text data, and the output is the sentiment analysis result. The server analyzes this result to determine how to adjust the meeting's progress.

[0616] Step 5:

[0617] Based on the sentiment analysis results, feedback is provided to the facilitator as needed. The input is the sentiment analysis results, and the output is a suggestion for adjusting the discussion. The terminal sends notifications to the facilitator, suggesting ways to promote the discussion or take a break.

[0618] Step 6:

[0619] After the meeting ends, the server automatically creates meeting minutes based on the generated text data. The input is the text data collected during the meeting, and the output is the meeting minutes. The meeting minutes, including sentiment analysis feedback, are delivered to the user.

[0620] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0621] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0622] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0623] [Fourth Embodiment]

[0624] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0625] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0626] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0627] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0628] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0629] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0630] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0631] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0632] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0633] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0634] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0635] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0636] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0637] This invention is a system aimed at improving the efficiency of meetings, which automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress during meetings, and creation and distribution of meeting minutes after the meeting.

[0638] First, the server collects users' calendar information and calculates common free time. Based on this information, it identifies mandatory participants according to their importance. For example, project leaders and key personnel will always be prioritized as meeting attendees.

[0639] Next, the server reserves an available meeting space during the acquired free time. This utilizes an interface with the company's internal meeting room reservation system. Details of the reserved meeting room are automatically added to the user's calendar.

[0640] During the meeting, the terminal uses speech recognition technology to transcribe speech in real time and sends the content to the server. The server monitors these speeches and has a function to notify users via the terminal if there is a deviation from the agenda. For example, if the content deviates from the scheduled agenda, it will warn the user with a message such as "You need to return to the agenda."

[0641] After the meeting ends, the server automatically generates meeting minutes using the text data collected during the meeting, summarizing key points as needed. These minutes and summaries are quickly distributed to relevant parties, facilitating information sharing.

[0642] Furthermore, users can provide feedback on their meeting experience through their devices. The server collects and analyzes this feedback to help improve future meetings.

[0643] Implementing this system can significantly reduce meeting preparation time and improve the efficiency of meeting management. For example, one company was able to halve the time spent on meeting-related tasks and improve employee productivity by introducing this system.

[0644] The following describes the processing flow.

[0645] Step 1:

[0646] The server retrieves users' calendar information and checks each user's schedule. Next, it compares these schedules and calculates common free time for everyone. Based on the importance of each user, it identifies those who must participate.

[0647] Step 2:

[0648] The server searches for available meeting locations based on the calculated free time. This is done through the meeting room reservation system. The server selects an available meeting room and automatically adds it to the user's calendar.

[0649] Step 3:

[0650] At the start of the meeting, the device activates its speech recognition function and transcribes the content spoken during the meeting into text in real time. The device then sends the recognized text to the server.

[0651] Step 4:

[0652] The server compares the received text against a predetermined agenda and checks the progress. If the discussion deviates from the agenda, the server notifies the user via the terminal that "the topic has deviated from the agenda."

[0653] Step 5:

[0654] The server monitors and manages the time allocated to each agenda item. When the scheduled time approaches, it sends an alert to the user via their terminal saying, "Please move on to the next agenda item."

[0655] Step 6:

[0656] After the meeting ends, the server automatically creates meeting minutes based on the text data generated during the meeting. These minutes and a summary are then sent to participants via email.

[0657] Step 7:

[0658] Users input feedback on the meeting via their devices. The server analyzes the collected feedback and considers ways to improve the next meeting.

[0659] (Example 1)

[0660] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0661] Meetings in companies and organizations are plagued by inefficiencies in preparation time and execution. It is necessary to reduce the time and effort involved in scheduling participants, securing meeting locations, managing the meeting, and creating and sharing minutes. Furthermore, it is essential to prevent meetings from deviating from the agenda and improve their quality. Additionally, an appropriate mechanism is needed to utilize participant feedback to improve future meetings.

[0662] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0663] In this invention, the server includes means for acquiring the activity schedules of multiple users and calculating shared free time; means for automatically reserving a meeting place based on the free time; and means for transcribing statements made during discussions using speech recognition and monitoring the progress. This enables automation of meeting preparation, highly accurate time management, efficient progress management, and rapid and accurate information sharing.

[0664] "User" refers to a person who uses this system to participate in a meeting.

[0665] "Planned activities" refers to a collection of schedule data that includes the date and time the user has planned to do so, as well as related event information.

[0666] "Shared free time" refers to common unscheduled time found by comparing the activity schedules of multiple users.

[0667] "Meeting place" refers to the location where the meeting is held, such as a conference room or an online meeting platform.

[0668] "Speech recognition" refers to the technology that converts spoken words during a discussion into text data in real time.

[0669] "Discussion" refers to an activity in which participants talk about an agenda item.

[0670] "Monitoring the progress" means that the server monitors the discussion in real time to ensure that it does not deviate from the set agenda and time allocation.

[0671] "Records" refer to documents and data compiled based on the content of discussions, and include summarized minutes as necessary.

[0672] A "generative artificial intelligence model" refers to an artificial intelligence system that uses generative AI models to perform summarization and information extraction based on input data.

[0673] "Key points" refer to information or content that should be given particular importance during a meeting or discussion.

[0674] In implementing this invention, the server collects user activity schedule data and calculates shared availability. Calendar services such as the Google Calendar API and Outlook Calendar API are utilized to obtain information from the user's terminal or calendar management application. Based on this information, the server optimizes the timing and automatically reserves the meeting place. General meeting room reservation management software is used in this process.

[0675] During the meeting, the terminal uses speech recognition technology to transcribe the user's speech in real time. This speech recognition process may utilize the Google Speech-to-Text API, among others. The generated text is sent to a server and monitored by a specific algorithm. This monitoring process incorporates a function to compare the text data with a pre-set agenda, and the user is notified if a deviation is detected.

[0676] After the meeting ends, the server automatically generates a record using the text data obtained during the meeting. By utilizing a generative artificial intelligence model, it is possible to summarize the meeting minutes and extract key points. One example of a model used here is OpenAI's GPT.

[0677] Furthermore, users can provide feedback on the meeting through their devices. This feedback data is collected and analyzed by the server and used to improve future meetings.

[0678] As a concrete example, one company reported that after implementing this system, employees were able to reduce the time spent on meeting-related tasks and use the freed-up time to complete projects, resulting in increased productivity.

[0679] Examples of prompts used to manipulate a generating AI model include instructions such as, "Summarize the key points of the meeting." Such instructions enable the generated AI model to quickly and accurately produce meeting minutes.

[0680] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0681] Step 1:

[0682] The server collects user activity schedule data through the terminal. This process uses the Google Calendar API and Outlook Calendar API to obtain calendar information. The input is the user's calendar information, and the output is a list of available time slots based on each participant's activity schedule. Based on this list, the server calculates common available time slots and determines the optimal meeting time.

[0683] Step 2:

[0684] The server automatically reserves meeting places based on common availability. Here, it uses the company's internal meeting room reservation system to reserve a meeting room suitable for the available time slot. The input is information about availability and available meeting rooms, and the output is detailed information about the reserved meeting room. This information is automatically added to the user's calendar.

[0685] Step 3:

[0686] During a meeting, the device uses speech recognition technology to transcribe the user's speech in real time. The Google Speech-to-Text API is commonly used, where speech data is input and text data is output. The generated text is sent to a server, and the content of the speech is recorded in real time.

[0687] Step 4:

[0688] The server monitors the received text data and compares it against the agenda. The input is the text data generated during the meeting, and the output is the monitoring results. If a participant's comments deviate from the agenda, the server notifies them via their terminal with the message, "You need to return to the agenda." This feedback function ensures the efficiency of the meeting.

[0689] Step 5:

[0690] After the meeting ends, the server uses a generative AI model to create a record. The input is all the text data generated during the meeting, and the output is a meeting transcript including a summary. Using OpenAI's GPT model, key points are extracted and summarized, and delivered quickly to the user.

[0691] Step 6:

[0692] Users provide feedback on their meeting experience through their devices. Input consists of user feedback comments, and output is analysis results to help improve future meetings. The server collects and analyzes this data to generate suggestions for improving the overall meeting experience.

[0693] (Application Example 1)

[0694] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] Online meetings and gatherings often present challenges such as the time-consuming process of coordinating participants' schedules and booking venues, as well as the difficulty of efficiently managing the meeting's progress and content. Furthermore, discussions can stray from the agenda, making progress management cumbersome. Additionally, creating meeting minutes afterward is time-consuming, often delaying information sharing among participants. Traditional systems have struggled to comprehensively address these issues.

[0696] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0697] In this invention, the server includes means for obtaining the activity schedules of multiple users and calculating common free time; means for automatically reserving a meeting place based on the free time; means for transcribing speech during a meeting into text using speech recognition and monitoring the progress; means for collecting feedback from users on the meeting to help improve future meetings; and means for displaying the progress of the agenda in real time using a visual assistance device. This reduces meeting preparation time, allows for proper management of the meeting's progress, and enables efficient minute-taking and rapid information sharing with participants.

[0698] A "user" is an individual or organization that uses this system to participate in meetings or gatherings.

[0699] "Activity schedule" refers to the schedules and events that users are planning to participate in.

[0700] "Free time" refers to the period of time that is available in common when considering the activity schedules of multiple users.

[0701] A "meeting place" refers to the physical or virtual location or environment where a meeting or event takes place.

[0702] "Speech recognition" is a technology that allows machines to understand spoken content and convert it into text data.

[0703] "Visual assistance devices" refer to smart glasses and other devices that are used to visually supplement information.

[0704] "Progress monitoring" is the process of managing the progress of a meeting or gathering and ensuring that it proceeds according to the scheduled agenda.

[0705] "Meeting minutes" are documents that record the content of discussions and decisions made at meetings or gatherings.

[0706] "Real-time display" means that information is presented to the user almost instantly, with virtually no time delay.

[0707] This system uses a cloud-based server to manage the activity schedules of multiple users. The server collects the users' activity schedules and calculates common free time from this data. Based on the calculated free time, the server automatically reserves meeting places. For example, if a virtual meeting room is needed, it reserves a virtual meeting room through the existing interface.

[0708] Furthermore, while the meeting is in progress, speech recognition technology is used to convert audio data from the terminal into text in real time. The Google Speech-to-Text API is used for this speech recognition. The text data is analyzed on the server using natural language processing libraries (e.g., spaCy or NLTK) to monitor whether the progress is on track with the agenda. If the progress deviates from the agenda, the server notifies the user via a visual aid.

[0709] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the text data acquired in real time. These minutes are immediately distributed to all relevant parties. Users can submit feedback on the meeting through this system, and this feedback is collected by the server and used to improve future meetings.

[0710] For example, one company implemented this system, allowing employees to use smart glasses to check the progress of meetings in real time and receive meeting minutes instantly. This enabled employees to improve meeting efficiency and increase productivity.

[0711] An example of a prompt is: "At the next meeting, please demonstrate a system that transcribes audio into text in real time, monitors the agenda progress, and automatically shares meeting minutes after the meeting."

[0712] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0713] Step 1:

[0714] The server retrieves users' activity schedules. It synchronizes with the calendar applications used by the users and imports each user's activity schedule data. The input is the user's calendar data, and the output is the basic data for calculating common free time. This process uses the calendar API to extract data.

[0715] Step 2:

[0716] The server analyzes the acquired activity schedule data and calculates common free time slots among multiple users. The input is the activity schedule data obtained in step 1, and the output is a list of time slots when all users are free. As part of the data processing, overlapping time slots are detected and common free time slots are listed.

[0717] Step 3:

[0718] The server automatically reserves meeting places based on common available time slots. The input is the available time data obtained in step 2, and the output is information about the reserved meeting places. As a data calculation, it selects meeting places that are available during the available time slots and performs the procedure using the reservation system API.

[0719] Step 4:

[0720] The device sends audio data from the meeting to a speech recognition service for real-time text conversion. The input is audio data acquired via the microphone, and the output is text data. This process involves analyzing the audio using a speech recognition API (e.g., Google Speech-to-Text).

[0721] Step 5:

[0722] The server analyzes the transcribed data and monitors the progress of the agenda. The input is the text data obtained in step 4, and the output is the analysis of the progress. Using a natural language processing library, the server compares the spoken content with the pre-set agenda to check for deviations from the agenda.

[0723] Step 6:

[0724] If a deviation from the progress is detected, the server will notify the user via a visual aid. The input is the progress analysis result obtained in step 5, and the output is a warning message. Specifically, a warning message is generated and displayed in real time on the user's visual aid.

[0725] Step 7:

[0726] After the meeting ends, the server automatically generates meeting minutes using a generative AI model based on the collected text data and distributes them to users. The input is the text data obtained in step 4, and the output is the automatically generated meeting minutes. This process extracts important statements and decisions and uses email or a dedicated system to quickly distribute them to users.

[0727] Step 8:

[0728] Users submit feedback about meetings to the system. The input is user feedback information, and the output is feedback data stored on the server. This feedback is used to improve future meetings, so the server collects it and incorporates it into the next meeting plan.

[0729] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0730] This invention is a system aimed at improving the efficiency of meetings, and a key feature is the incorporation of an emotion engine. This system automatically handles participant scheduling, automatic booking of meeting venues, monitoring of meeting progress, and creation and distribution of meeting minutes after the meeting, while also recognizing and analyzing participants' emotions to aid in the smooth running of the meeting.

[0731] First, the server retrieves users' calendar information and analyzes each user's schedule to identify common free time slots. Then, it automatically reserves a suitable meeting location for those free time slots and registers it in the user's calendar. The emotion engine activates as soon as the meeting begins, monitoring the emotional state of the participants.

[0732] The terminal uses speech recognition technology to transcribe speech in real time during the meeting and sends the content to the server. The server monitors the progress of the agenda and analyzes the emotional state of the participants. If a participant's emotions change in a negative direction, the server notifies the meeting facilitator through the terminal and adjusts the meeting's progress as needed.

[0733] During the meeting, the server manages the time allocated to each agenda item and optimizes the process based on sentiment analysis results. For example, if participants are feeling tired even though the agenda is progressing as planned, the server will send a notification suggesting a short break.

[0734] After the meeting, the server automatically creates meeting minutes based on the generated text data and summarizes the results of the emotional state analysis. These minutes and summaries are distributed to participants to help them adjust the strategy for the next meeting based on their feedback.

[0735] As a concrete example, by implementing this system in one company, they were able to improve customer satisfaction by maintaining participants' concentration during meetings and appropriately managing their emotional states. Thus, the present invention provides a comprehensive solution for improving the productivity of meetings.

[0736] The following describes the processing flow.

[0737] Step 1:

[0738] The server retrieves users' calendar information via an API and analyzes each user's schedule. This allows it to calculate common free time for all participants. It then identifies mandatory participants based on their importance.

[0739] Step 2:

[0740] The server checks the company's internal meeting room reservation system for available meeting spaces based on shared availability. It automatically reserves the selected meeting room and reflects that information in each user's calendar.

[0741] Step 3:

[0742] At the start of the meeting, the terminal activates its speech recognition function and transcribes all statements made during the meeting into text in real time. This text information is sent to a server and used for progress management.

[0743] Step 4:

[0744] The device uses its built-in emotion engine to recognize each user's emotions from their voice and speech recognition results. The identified emotion information is then sent to the server.

[0745] Step 5:

[0746] The server analyzes text and emotional information obtained from the audio and monitors the progress of the agenda. If the user's emotional state is negative, the server sends a notification to the facilitator via the terminal to draw their attention.

[0747] Step 6:

[0748] During the meeting, the server measures the time elapsed for each agenda item and sends an alert to users via their terminals if the scheduled time is likely to be exceeded. Based on the emotion engine data, it recommends adjusting the progress as needed.

[0749] Step 7:

[0750] After the meeting ends, the server automatically generates meeting minutes based on the speech recognition results. If there are any notable emotional shifts, a summary including those will also be created. The minutes will be distributed to all participants.

[0751] Step 8:

[0752] Users can provide feedback on meetings through their devices. The server uses the collected feedback and sentiment analysis results to formulate suggestions for improving the next meeting.

[0753] (Example 2)

[0754] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] In today's business environment, meetings play a crucial role in corporate operations, yet many are inefficient and contribute to decreased productivity. Improving the efficiency of meetings requires proper adjustment of many factors, including time management, location selection, progress monitoring, and participant emotional state management. However, traditional systems have struggled to comprehensively and automatically manage these factors.

[0756] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0757] In this invention, the server includes means for acquiring time management data from multiple information users and calculating common free time, means for automatically reserving a meeting place based on the free time, and means for documenting speech during the meeting using speech conversion technology and monitoring its progress. This improves the efficiency of the meeting and enables the operation of a highly productive meeting by managing the emotional state of the participants.

[0758] "Information users" refers to individual users who use the system and participate in the management or participation of a gathering.

[0759] "Time management data" refers to schedule information, including the user's appointments and free time.

[0760] "Shared free time" refers to time slots that are free at the same time on the schedules of multiple information users.

[0761] A "meeting place" refers to a physical or virtual location where a gathering takes place.

[0762] "Voice conversion technology" refers to the technology that converts spoken audio from a group into text data in real time.

[0763] "Means of monitoring progress" refers to a function that tracks the progress of a meeting in real time and makes adjustments as needed.

[0764] "Emotional analysis" refers to a technology that detects and analyzes the emotional state of participants.

[0765] "Meeting records" refer to documents that transcribe the statements and decisions made during a meeting into text.

[0766] "Means of gathering opinions" refers to the function of collecting feedback and suggestions from information users regarding the group.

[0767] This invention is a comprehensive system designed to enable information users to efficiently manage gatherings. Its main features include the high degree of automation of elements such as time management, location selection, progress monitoring, sentiment analysis, and opinion gathering.

[0768] The server utilizes various calendar APIs to obtain time management data from information users. These APIs collect information users' schedules and function as a database for calculating common free time. A specific example of such an API is the time management system API.

[0769] The server automatically reserves meeting places based on common availability. For example, it uses a reservation system API to check the availability of physical meeting rooms or online platforms and complete reservations.

[0770] The terminal uses speech conversion technology to document conversations in real time. Specific technologies that could be used include speech recognition software libraries and speech recognition APIs. This allows for rapid text conversion and transmission to a server.

[0771] The server monitors the progress of the meeting based on speech-recognized documents and uses an emotion analysis engine to analyze participants' emotional states in real time. This helps to provide appropriate advice to the meeting facilitator.

[0772] After the meeting concludes, the server distributes a meeting record generated by a generative AI model using speech recognition text to information users and collects feedback and opinions from participants. Natural language generation models can be used as an example of generative AI model usage.

[0773] In one specific case, implementing this system within a company resulted in a 30% reduction in typical meeting times while simultaneously lowering participant fatigue. This led to smoother workflow and improved labor productivity.

[0774] As an example of a prompt, the AI ​​model can be given the instruction, "Summarize the following meeting content and propose a strategy for the next meeting based on the opinions submitted by the participants."

[0775] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0776] Step 1:

[0777] The server retrieves user time management data. The input is each user's schedule information. Specifically, it collects appointments from the user's calendar using a time management system API. The collected schedule information is analyzed to identify common free time periods for each user. The output is a list of common free time periods that could potentially form a group.

[0778] Step 2:

[0779] The server takes common available time slots as input and automatically reserves meeting places based on that availability. It checks for available meeting places at specific dates and times via the reservation system API and automatically confirms reservations based on availability. The output is a reservation completion notification, which is reflected in the user's calendar.

[0780] Step 3:

[0781] Once the meeting begins, the terminal uses speech recognition technology to document speech in real time. It receives audio data as input and converts it to text data using a speech recognition software library. The output is the text of the speech to be recorded.

[0782] Step 4:

[0783] The server receives text data of speech sent from terminals during the meeting and monitors its progress. It analyzes this data to track the progress of the agenda and the content of participants' comments. Furthermore, it uses a sentiment analysis engine to evaluate participants' emotional states in real time. If problems are detected, it notifies the meeting facilitator. The output includes a progress monitoring report and sentiment analysis results.

[0784] Step 5:

[0785] After the meeting concludes, the server automatically generates a meeting record using an AI model based on the accumulated spoken text data. The output is an editable meeting record, which is then distributed to all participants. The generated record is distributed in a format that includes prompts for collecting feedback from participants.

[0786] Step 6:

[0787] Users provide their opinions and feedback to the server after the meeting using the received meeting records. This feedback data is used as reference material for considering improvements in preparation for the next meeting. The output is a collection of materials for optimizing the planning of the next meeting.

[0788] (Application Example 2)

[0789] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0790] In meetings, accurately understanding participants' emotional states and making appropriate adjustments accordingly is difficult. This leads to decreased meeting efficiency and increased participant stress. Furthermore, traditional meeting management systems cannot provide real-time feedback based on participants' emotions, thus failing to improve the quality of meetings.

[0791] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring the schedules of multiple users and calculating common free time, means for automatically reserving meeting locations based on free time, means for transcribing speech during a meeting into text using speech recognition and monitoring the progress, and means for analyzing the emotional state of users and adjusting the progress of the meeting based on the analysis results. This makes it possible to accurately grasp the emotions of participants during a meeting and optimize the progress based on that.

[0792] A "user" refers to an individual who utilizes the system, and is the entity that obtains schedules and participates in meetings.

[0793] "Schedule" refers to date and time information related to activities and events set by the user, and is used for scheduling meetings.

[0794] "Free time" refers to the period of time when no appointments are scheduled for any of the users, as determined by comparing their schedules.

[0795] The term "meeting location" is a concept that includes not only the physical place selected for holding a meeting, but also, in some cases, a virtual location.

[0796] "Speech recognition" is a technology that converts spoken words during a meeting into text data in real time.

[0797] "Facilitation" refers to management activities, including the scheduled agenda and time management of a meeting.

[0798] "Emotional state" is an indicator that shows the psychological and emotional condition of participants, and is used as a subject of analysis to revitalize and improve meetings.

[0799] "Analysis results" refer to the information and data obtained after analyzing emotional states, and serve as the basis for adjusting the meeting's progress.

[0800] "Analysis" refers to the act of analyzing a user's emotional state and understanding their condition and trends.

[0801] The system for carrying out this invention consists of a server, a user terminal, and associated software. The server first retrieves the schedules of multiple users, analyzes the data to identify common free time slots, and then automatically selects and reserves the optimal meeting location based on the free time. When a meeting begins, the user terminal uses speech recognition technology to transcribe the spoken words in the meeting in real time and sends the text to the server.

[0802] The server monitors the progress of the meeting based on the acquired text information and simultaneously analyzes the emotional state of the users. Using a generative AI model, it analyzes the participants' psychological state in real time and optimizes the meeting's progress based on the results. For example, if the content of a discussion deviates from the agenda or if a user's emotions are leaning towards the negative, the server sends a notification to the facilitator's terminal to prompt necessary adjustments.

[0803] Specifically, if the sentiment analysis indicates that a participant is feeling tired, feedback such as suggesting a short break can be provided. Furthermore, after the meeting, the server automatically creates meeting minutes based on the generated text data and distributes them to all participants. The minutes will also include the sentiment analysis feedback, which can be used to improve future meetings and forums.

[0804] The specific hardware and software used include schedule management using the Google Calendar API, speech-to-text conversion using Amazon Polly, and sentiment analysis using the OpenAI API.

[0805] As a concrete example, using this system in a community discussion meeting allows for real-time monitoring of each participant's opinions and emotional state, enabling efficient and positive progress. The facilitator can use smart devices to encourage active discussion and adjust the agenda in a timely manner.

[0806] Examples of prompt statements for a generative AI model are as follows:

[0807] "Analyze the emotions in the following conversation and identify possible emotional states: Meeting transcript text."

[0808] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0809] Step 1:

[0810] The server retrieves each user's schedule via the Google Calendar API. The input is the user's calendar information, and the output is common free time data. The server then calculates and identifies the time slots when all users are available to participate.

[0811] Step 2:

[0812] The server automatically selects the optimal meeting location and makes a reservation based on availability data. The input is the specified availability time, and the output is reservation information. This information is registered as a calendar event on the user's device.

[0813] Step 3:

[0814] After the meeting begins, the user's device uses speech recognition technology (Amazon Polly) to transcribe the meeting's speech into text in real time. The input is the audio from the meeting, and the output is text data. This data is sent to a server and serves as basic information for monitoring the meeting's progress.

[0815] Step 4:

[0816] The server uses an API (OpenAI) for a generative AI model to analyze the sentiment of meeting participants based on the acquired text data. The input is text data, and the output is the sentiment analysis result. The server analyzes this result to determine how to adjust the meeting's progress.

[0817] Step 5:

[0818] Based on the sentiment analysis results, feedback is provided to the facilitator as needed. The input is the sentiment analysis results, and the output is a suggestion for adjusting the discussion. The terminal sends notifications to the facilitator, suggesting ways to promote the discussion or take a break.

[0819] Step 6:

[0820] After the meeting ends, the server automatically creates meeting minutes based on the generated text data. The input is the text data collected during the meeting, and the output is the meeting minutes. The meeting minutes, including sentiment analysis feedback, are delivered to the user.

[0821] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0822] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0823] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0824] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0825] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0826] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0827] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0828] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0829] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0830] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0831] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0832] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0833] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0834] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0835] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0836] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0837] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0838] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0839] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0840] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0841] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0842] The following is further disclosed regarding the embodiments described above.

[0843] (Claim 1)

[0844] A method for obtaining the schedules of multiple users and calculating common free time,

[0845] A method for automatically booking meeting venues based on availability,

[0846] A means of using speech recognition to transcribe speech during a meeting into text and to monitor the progress of the meeting,

[0847] A method for automatically generating meeting minutes and distributing them to users,

[0848] A means of collecting feedback from users about the meeting and using it to improve the next meeting,

[0849] A system that includes this.

[0850] (Claim 2)

[0851] The system according to claim 1, comprising means for comparing the content of a statement obtained by speech recognition with the agenda and notifying of a deviation from the agenda.

[0852] (Claim 3)

[0853] The system according to claim 1, comprising means for monitoring the time occupied and notifying the user if the scheduled time is exceeded.

[0854] "Example 1"

[0855] (Claim 1)

[0856] A means of obtaining the activity schedules of multiple users and calculating shared free time,

[0857] A method for automatically booking a meeting place based on available time,

[0858] A means of transcribing speech during a discussion using speech recognition and monitoring its progress,

[0859] A means of automatically generating and distributing records to users,

[0860] A means of collecting feedback from users on the discussion and using it to improve the next discussion,

[0861] A means of matching the content of speech acquired through speech recognition with the agenda and notifying the user of any deviation from the agenda,

[0862] A system that includes this.

[0863] (Claim 2)

[0864] The system according to claim 1, comprising means for monitoring the time occupied and notifying the user of any exceedance of the scheduled time.

[0865] (Claim 3)

[0866] The system according to claim 1, comprising means for extracting and summarizing key points using a generation artificial intelligence model when generating records.

[0867] "Application Example 1"

[0868] (Claim 1)

[0869] A method for obtaining the activity schedules of multiple users and calculating common free time,

[0870] A method for automatically booking a meeting place based on available time,

[0871] A means of using speech recognition to transcribe speech during a meeting into text and to monitor its progress,

[0872] A method for automatically generating meeting minutes and distributing them to users,

[0873] A means of collecting feedback from users about the meeting and using it to improve the next meeting,

[0874] A means of displaying the progress of the agenda in real time using a visual aid device,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, comprising means for comparing the content of a statement obtained by speech recognition with the agenda and notifying of a deviation from the agenda.

[0878] (Claim 3)

[0879] The system according to claim 1, comprising means for monitoring the time occupied and notifying the user if the scheduled time is exceeded.

[0880] "Example 2 of combining an emotion engine"

[0881] (Claim 1)

[0882] A means of obtaining time management data from multiple information users and calculating common free time,

[0883] A method for automatically booking a meeting place based on available time,

[0884] A means of documenting speech during a meeting using speech conversion technology and monitoring its progress,

[0885] A means of performing emotional analysis and optimizing the progress of a group,

[0886] A means of automatically collecting and generating records and distributing them to information users,

[0887] A means of collecting feedback from information users about the gathering and using it to improve the next gathering,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, comprising means for comparing the content of a statement obtained by voice conversion with the agenda and notifying of a deviation from the agenda.

[0891] (Claim 3)

[0892] The system according to claim 1, comprising means for monitoring the time occupied and notifying the information user of any exceeding of the scheduled time.

[0893] "Application example 2 when combining with an emotional engine"

[0894] (Claim 1)

[0895] A method for obtaining the schedules of multiple users and calculating common free time,

[0896] A means of automatically reserving meeting locations based on availability,

[0897] A means of using speech recognition to transcribe speech during a meeting into text and to monitor the progress of the meeting,

[0898] A method for automatically generating meeting minutes and distributing them to users,

[0899] A means of collecting user feedback on meetings and using it to improve future meetings,

[0900] A means of analyzing the emotional state of users and adjusting the meeting proceedings based on the analysis results,

[0901] A system that includes this.

[0902] (Claim 2)

[0903] The system according to claim 1, comprising means for comparing the content of a statement obtained by speech recognition with the agenda and notifying of a deviation from the agenda.

[0904] (Claim 3)

[0905] The system according to claim 1, comprising means for monitoring the time occupied and notifying the user if the scheduled time is exceeded. [Explanation of symbols]

[0906] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A method for obtaining the activity schedules of multiple users and calculating common free time, A method for automatically booking a meeting place based on available time, A means of using speech recognition to transcribe speech during a meeting into text and to monitor its progress, A method for automatically generating meeting minutes and distributing them to users, A means of collecting feedback from users about the meeting and using it to improve the next meeting, A means of displaying the progress of the agenda in real time using a visual aid device, A system that includes this.

2. The system according to claim 1, further comprising means for comparing the content of a statement obtained by speech recognition with the agenda and notifying of any deviation from the agenda.

3. The system according to claim 1, comprising means for monitoring the time occupied and notifying the user if the scheduled time is exceeded.

Images