system

Abstract

Provide a system. 【Solution means】 Means to obtain the schedule information of participants and propose an optimal meeting time, Means to automatically generate items for the meeting based on the purpose and communicate them to the participants, Means to analyze the voice during the meeting and convert the speech into text in real time, Means to automatically generate a record based on the speech content and share it with the participants after the meeting, Means to organize the action items determined in the meeting and notify the person in charge of an individual work list, Means to automatically adjust the date and time of the next meeting and propose it to the participants, Means to analyze the meeting data and make improvement proposals based on the speech data of the participants, Means to link the status of the facilities and the maintenance activity schedule and propose an optimal maintenance activity time, Means to automatically place orders for necessary equipment and parts, A system including.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a modern business environment, there is a problem that many meetings are conducted inefficiently, wasting the time and resources of participants. Specifically, it often takes too much time to schedule meetings, and participants often get lost in discussions that are not based on the agenda. There is also a problem that it is difficult to accurately record the content of meetings and efficiently manage the decisions and next steps. Furthermore, there is a lack of a mechanism to appropriately evaluate the speeches and participation of people attending meetings and use it to improve the next meeting. It is an object of the present invention to solve such problems.

Means for Solving the Problems

[0005] This invention provides a system that improves the overall efficiency of meetings by utilizing an AI agent. Specifically, it includes means for automatically analyzing participants' meeting schedules and proposing optimal meeting times. It also includes means for distributing agendas automatically generated based on the purpose of the meeting to participants. Furthermore, it provides a system that uses speech recognition technology to transcribe speech in real time during meetings and automatically generates meeting minutes based on this transcription. Subsequently, it has a function to automatically notify the person in charge of the action items decided during the meeting. When scheduling the next meeting, it also proposes optimized candidate dates and times based on feedback from participants. Finally, by analyzing speech data during meetings and providing suggestions for improvement regarding participant speaking frequency and the progress of discussions, it is possible to improve the efficiency and productivity of meetings.

[0006] "Participants" refers to individuals or members of organizations who attend a meeting and participate in its discussions.

[0007] "Schedule information" refers to data on events and free time recorded in participants' calendars or schedule management systems.

[0008] "Meeting time" refers to the time period from the start to the end of a meeting, including the time when all participants are scheduled to be present.

[0009] An "agenda" refers to a document that systematically lists the topics and items that are scheduled to be discussed at a meeting.

[0010] "Analyzing audio" refers to the process of converting audio data spoken during a meeting into text format using speech recognition technology.

[0011] "Text conversion" refers to the process of converting audio or image data into digital data in the form of text.

[0012] "Meeting minutes" refers to a document that records the content of discussions and decisions made during a meeting.

[0013] "Action items" refer to specific actions or tasks decided upon during a meeting, and usually include a person in charge and a deadline.

[0014] "Feedback" refers to opinions, requests, and evaluations of the meeting from participants regarding the system, and the system is improved based on this feedback.

[0015] "Statement data" refers to digital data that records the words and content spoken by participants during a meeting.

[0016] "Improvement suggestions" refer to specific proposals for making meeting management and discussions more effective. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The meeting support system according to the present invention is designed to embody a series of processes for efficiently operating and managing meetings throughout the pre-meeting preparation, during the meeting, and after the meeting. Its embodiments are described in detail below.

[0039] The core of this system lies in a server, which collects data from the APIs of the scheduling management systems used by each participant to obtain their schedule information. The server analyzes the collected data and automatically suggests the optimal meeting time by comparing the availability of all participants.

[0040] The server then automatically generates an agenda based on the meeting's objectives. It utilizes past meeting records and purpose-specific templates to set appropriate topics. This generated agenda is then communicated to participants via email or a dedicated application, allowing them to review it before the meeting.

[0041] During the meeting, the server uses speech recognition technology to transcribe participants' statements in real time. The server analyzes the audio data, identifies speakers, and automatically generates meeting minutes. This feature allows participants to focus on the discussion and reduces the effort required for record-keeping.

[0042] After the meeting, the server automatically generates meeting minutes, summarizes them, extracts key points, and distributes them to participants. It also organizes the action items decided during the meeting and notifies the responsible parties with specific deadlines. During this process, the terminal displays a to-do list to the user, allowing them to manage their progress.

[0043] Furthermore, when scheduling the next meeting, the server collects feedback from participants and, based on that, proposes the most suitable dates and times again. Iterating through this process promotes further efficiency.

[0044] The data from meetings is analyzed in detail by a server, recording things like the frequency of each speaker and the flow of topics during the meeting. Based on this data, points for improvement in future meetings are suggested. These suggestions are distributed to each participant as a dedicated report, serving as foundational material for improving the quality of meetings.

[0045] For example, if the server sets the meeting theme to "Market launch of a new product," it will automatically generate an agenda aligned with that theme and send it to the participants. As the meeting progresses, the server will record each statement and guide the participants to efficiently carry out the next steps.

[0046] In this way, servers, terminals, and users work together within the entire system, enabling efficient meeting management. In particular, the server's data analysis and optimization algorithms contribute to simplifying meeting preparation and reducing the burden on participants.

[0047] The following describes the processing flow.

[0048] Step 1:

[0049] The server retrieves schedule information from participants' schedule management systems via API. All participants' schedules are stored in a database and prepared for analysis.

[0050] Step 2:

[0051] The server compares each participant's schedule and extracts common available time slots. This allows it to list the optimal meeting times that all participants can attend.

[0052] Step 3:

[0053] The server automatically generates an agenda based on the meeting's objectives, using past meeting records and preset templates. The generated agenda is then sent to all participants via email or a dedicated app.

[0054] Step 4:

[0055] When a meeting begins, the terminal sends the meeting audio to the server. The server uses a speech recognition engine to transcribe the audio into text in real time, creating the basic data for the meeting minutes.

[0056] Step 5:

[0057] The server analyzes the transcribed data, organizes the content for each speaker, and incorporates it into the meeting minutes. Simultaneously, it monitors the progress of the meeting and manages time according to the agenda. It issues instructions to move on to the next topic as needed.

[0058] Step 6:

[0059] After the meeting ends, the server automatically summarizes the generated minutes and extracts key decisions and to-do lists. Based on this, it sends a report to participants via email.

[0060] Step 7:

[0061] The server organizes the action items decided during the meeting by person in charge, sets specific deadlines, and notifies each participant via their terminal as a to-do list.

[0062] Step 8:

[0063] The server adjusts the next meeting date based on feedback collected from participants. It presents participants with proposed dates and times to facilitate the scheduling process.

[0064] Step 9:

[0065] The server analyzes the data collected during the meeting to evaluate the frequency of participation and the progress of the discussion. Based on this analysis, improvement suggestions are compiled into a report and provided to the participants.

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] In today's business environment, efficient meeting management is a critical challenge. Especially when many participants are involved, significant effort is required to optimize meeting times, create agendas, and record and share meeting content. Furthermore, it's essential to properly manage the action items decided upon after the meeting and use them to improve future meetings. Traditional systems fail to efficiently address these challenges, making it difficult to improve the quality of meetings.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes means for acquiring participants' schedule information and suggesting the optimal time, means for automatically generating an agenda based on the purpose of the meeting and distributing it to participants, and means for analyzing speech and transcribing speech into text in real time. As a result, meeting preparation time is significantly reduced, participants can concentrate on the discussion, post-meeting action item management is streamlined, and the overall quality of the meeting can be improved.

[0071] "Participant schedule information" refers to data on the schedules and available times of individuals participating in the meeting.

[0072] "Suggesting the optimal time" is the process of selecting the most appropriate and efficient meeting start time, taking into account the schedules of all participants.

[0073] A "meeting agenda based on the purpose of the meeting" is a list of topics and subjects set in line with the focus and goals to be achieved at the meeting.

[0074] "Automatic generation" refers to a system creating content based on predetermined algorithms and data, without manual human intervention.

[0075] "Analyzing speech and converting it to text in real time" refers to the process of instantly converting spoken words during a meeting into text information using speech recognition technology.

[0076] "Automatically generating records" means that the system independently organizes the collected data and creates records based on that data.

[0077] A "list of individual tasks" is a list that summarizes the tasks and responsibilities decided during the meeting for each participant.

[0078] "Automatically adjusting the next meeting date" is a process that re-examines the availability of meeting participants and suggests the optimal date and time for the next meeting.

[0079] "Analyzing data and making improvement suggestions based on participants' comments" means analyzing recorded comment data, identifying areas for improvement in meeting management methods, and proposing solutions.

[0080] "Generating a document describing technical features" involves creating a document that summarizes the meeting content from a technical perspective, based on meeting records and key points.

[0081] This conference support system is designed to facilitate efficient communication between servers, terminals, and users. Specific implementations using hardware and software are described below.

[0082] hardware

[0083] The server retrieves participants' schedule information using the APIs of each scheduling management system. Specifically, it collects each participant's available time using the Google® Calendar API and the Microsoft® Graph API. The server uses a high-performance server computing unit suitable for processing large amounts of data.

[0084] software

[0085] The system software running on the server analyzes the acquired schedule data and executes an algorithm to calculate the optimal meeting time. For speech recognition technology, Azure® Speech Service and Google Speech-to-Text API are used. This technology transcribes speech during meetings into text in real time and automatically generates meeting minutes based on the content of the speech.

[0086] The server automatically generates an agenda based on the meeting's objectives using a generation AI model. This utilizes a template-based generation AI model, allowing the AI ​​to suggest appropriate agenda items based on prompts such as "Please create an agenda for the market launch of a new product."

[0087] Usage example

[0088] Users can access the platform from their devices and view meeting details. For example, the server's suggested dates and times for the next meeting are generated based on user feedback. With a prompt such as, "Please tell me the best meeting time considering everyone's availability," the server uses AI to suggest the most suitable options.

[0089] This system streamlines the entire process, from meeting preparation and execution to scheduling the next meeting, allowing users to focus more on the discussion. Data generated at each stage of the process is continuously analyzed by the server, contributing to improved quality in subsequent meetings. In this way, it is possible to achieve both increased efficiency and improved quality throughout the entire meeting process.

[0090] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0091] Step 1:

[0092] The server retrieves schedule information from the API of the scheduling management system used by each participant. This input data contains information about each participant's schedule. The server collects this data and stores it in a database to list the free time of all participants. As part of the data processing, it runs an algorithm to remove overlapping time and non-work time.

[0093] Step 2:

[0094] The server compares and analyzes the availability of all participants based on the stored schedule data. The input data is the schedule information processed in Step 1. The server calculates and proposes the optimal meeting time. In this process, a generative AI model is used, and prompts such as "Please tell me the optimal meeting time considering the availability of all participants" are used to have the AI ​​calculate the recommended time. The optimal meeting time is generated as output.

[0095] Step 3:

[0096] The server receives the meeting objective and automatically generates a corresponding agenda using an AI model. The input at this stage is information about the meeting objective. It selects and constructs appropriate topics and subjects using past meeting records and templates. The output is the automatically generated agenda distributed to participants. A specific example of its operation is using the prompt, "Please create agenda items regarding the market launch of the new product."

[0097] Step 4:

[0098] During the meeting, the server captures audio in real time and uses speech recognition technology to transcribe the speech into text. The input data is the audio data collected during the meeting. The audio data is analyzed, separated by speaker, and output as text data. The system operates by using a speech recognition API (e.g., Azure Speech Service) to perform simultaneous text transcription.

[0099] Step 5:

[0100] The server automatically generates meeting minutes based on text data obtained through speech recognition. The input data is the text data generated in step 4. The server summarizes this data, extracts key points, and structures it as meeting minutes. The output is the meeting minutes, which are shared after the meeting. The server has the function to distribute the meeting minutes electronically to participants.

[0101] Step 6:

[0102] The server organizes the action items decided during the meeting and creates a task list for each person in charge. This uses the meeting minutes created in step 5 as input. The action items are notified along with the person in charge and the deadline, and the output is an individual task list. The terminal displays this task list to the user, allowing them to track and check the progress.

[0103] Step 7:

[0104] The server collects feedback from participants to optimize the date and time of the next meeting. The input data is feedback collected from participants. Based on this, a generative AI model and the prompt "Please tell me the best date and time for the next meeting" are used to calculate and suggest candidate dates and times. The output is the adjusted candidate date and time for the next meeting.

[0105] Step 8:

[0106] The server meticulously analyzes the speech data collected during and after the meeting. The input data consists of speech data from all meeting participants. The server analyzes the frequency of speech and the flow of discussion to identify areas for improvement. The output is provided to participants as a report to help improve the quality of future meetings. A data analysis algorithm is used in its operation.

[0107] (Application Example 1)

[0108] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0109] While efficiency improvements are needed in data center and other facility operations, such as meetings and maintenance, coordinating participant schedules, creating agendas, and managing meeting minutes still require considerable time and effort. Furthermore, facility maintenance and procurement of necessary parts are predominantly managed manually, often leading to inefficiencies. In this context, there is a need to achieve efficiency and automation across the entire process of meeting management and facility management.

[0110] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0111] In this invention, the server includes means for acquiring participants' schedule information and proposing the optimal meeting time; means for automatically generating meeting items based on the purpose and communicating them to participants; means for analyzing audio during the meeting and transcribing speech in real time; means for automatically generating a record based on the content of speech and sharing it with participants after the meeting; means for linking the status of equipment with maintenance activity schedules and proposing the optimal timing for maintenance activities; and means for automatically ordering necessary equipment and parts. This reduces the burden of meeting management and enables efficient equipment management.

[0112] "Participants" are people who are involved in a meeting or activity and who have the role of providing schedules or opinions.

[0113] "Meeting time" refers to the time when participants gather to exchange information and make decisions.

[0114] An "agenda" is a list of topics and content to be addressed during a meeting, and it indicates the purpose and direction of the conference.

[0115] "Speech analysis" refers to the technology that acquires spoken words as audio data and converts it into text information.

[0116] "Transcribing speech" is the process of recording spoken content as digital text.

[0117] "Records" refer to documents or data that preserve the content of discussions during a meeting in a way that allows for later reference.

[0118] "Maintenance activities" refer to periodic maintenance and repair activities carried out to maintain the normal operation of equipment and systems.

[0119] "Equipment and parts" refers to the physical elements and replaceable parts necessary for the operation of the equipment.

[0120] "Automated ordering" is a process that orders necessary goods based on predetermined conditions, with minimal human involvement.

[0121] This system is built around a multi-functional server. The server utilizes cloud services, such as AWS® Lambda and DynamoDB, to efficiently process data. The server can retrieve participants' schedule information from the Google Calendar API, analyze it, and automatically suggest the optimal meeting time. This information is then distributed to participants via smartphones and computers.

[0122] Furthermore, the server uses generative AI models such as OpenAI's GPT-3 to generate an agenda based on the purpose of the meeting. This makes it possible to notify participants of the meeting's flow in advance and encourage them to prepare. During the meeting, Google Cloud Speech-to-Text is used to transcribe speeches in real time, and this data is stored in DynamoDB.

[0123] Furthermore, after the meeting, the server summarizes key points based on accumulated data and individually notifies participants of specific action items. The server also calculates the optimal timing for maintenance activities based on the equipment status and maintenance schedule, and automatically places orders for necessary equipment and parts in advance.

[0124] For example, in a maintenance meeting for cooling systems within a data center, the server can suggest the optimal repair time and place orders considering parts inventory. A possible prompt request might be, "Please suggest the optimal date for the next cooling system repair, taking into account the team members' schedules."

[0125] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0126] Step 1:

[0127] The server retrieves participants' schedule information via the Google Calendar API. The input is each participant's calendar access permission, and the server collects their schedule data based on this. The output is schedule data including the free time slots of all participants. Based on this, the server calculates the intersection of free time slots and proposes the optimal meeting time.

[0128] Step 2:

[0129] The server uses the collected data and leverages OpenAI's GPT-3 generative AI model to automatically generate an agenda based on the meeting's objectives. Inputs include the meeting's theme and past meeting minutes. Output is a detailed meeting agenda sent to participants. The server distributes this agenda to devices via email or a dedicated application.

[0130] Step 3:

[0131] During the meeting, the server uses Google Cloud Speech-to-Text to transcribe participants' speech in real time. The input is audio data collected during the meeting, and the output is transcribed text data. The server stores this data in DynamoDB for later processing.

[0132] Step 4:

[0133] After the meeting concludes, the server analyzes the accumulated text data and summarizes the key points. The input is the transcribed and saved meeting minutes, and the output is the summarized meeting minutes. The server uses this summary to send each participant a to-do list of action items.

[0134] Step 5:

[0135] The server analyzes equipment status and maintenance logs to plan the optimal timing for maintenance activities. Inputs include equipment operating status data and repair history data. Outputs are specific activity schedules notified to the maintenance team. Furthermore, the server predicts necessary equipment and parts and places purchase orders through an automated ordering system.

[0136] Step 6:

[0137] The user uses prompts to send details about the next meeting and other specific requests to the generating AI model. The input is a text-based request from the user (e.g., "Please suggest the best date for the next cooling system repair, taking into account the team members' schedules"). The output is the optimal suggestion generated by the AI ​​and displayed on the user's device.

[0138] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0139] This invention is a system designed to improve the quality of meetings by not only streamlining meeting management but also analyzing the emotional state of participants. This system utilizes artificial intelligence, combining speech recognition and emotion analysis to grasp participants' emotions in real time and provide feedback to facilitate the meeting.

[0140] The central server of the system receives audio data from meeting participants and first uses a speech recognition engine to transcribe their speech into text in real time. Simultaneously, an emotion engine analyzes speech patterns and linguistic expressions to detect emotional states. For example, it can identify positive, negative, or neutral emotions based on the tone and speed of speech, as well as specific keywords.

[0141] By performing sentiment analysis in real time, the server sends instructions to the terminal to appropriately adjust the progress of the meeting. The terminal can then notify the user, for example, "Participant A may be losing interest" or "The agenda item seems to have suddenly elicited a negative reaction," prompting adjustments to the ongoing agenda.

[0142] After the meeting, the server analyzes the data obtained by the emotion engine and generates a detailed report summarizing emotional trends and responses to specific topics, which is then distributed to participants. This provides material for concretely considering improvements for the next meeting and supports better decision-making.

[0143] For example, if participants' emotions regarding a particular product topic show a generally negative trend during a meeting, the emotional data will be used to consider areas for product improvement or switching to a different topic. This feedback will be provided after the meeting and used to prepare for the next meeting.

[0144] This system enables the server to manage meetings dynamically based on emotional data, and makes it easier for users to grasp the overall atmosphere of the meeting. This increases the likelihood of improved satisfaction for all participants and increased meeting productivity.

[0145] The following describes the processing flow.

[0146] Step 1:

[0147] The server accesses each participant's calendar API to retrieve their schedule information and collects their appointments. Based on this, it identifies the optimal meeting time and proposes it to all participants.

[0148] Step 2:

[0149] The server automatically generates an agenda based on the meeting's objectives. The generated agenda is then distributed to participants via email or a dedicated application.

[0150] Step 3:

[0151] Once the meeting begins, the terminal sends the audio data from the meeting to the server. The server uses a speech recognition engine to transcribe the spoken words into text in real time.

[0152] Step 4:

[0153] The server has an emotion engine built in that analyzes audio data to identify the emotional state of participants. It assigns positive, negative, and neutral emotion tags.

[0154] Step 5:

[0155] Based on the results of sentiment analysis, the server evaluates the progress of the meeting in real time and suggests adjusting the agenda to the user via the terminal at the appropriate time.

[0156] Step 6:

[0157] After the meeting ends, the server generates meeting minutes and comprehensively analyzes participants' reactions based on sentiment data. This then generates a detailed sentiment report.

[0158] Step 7:

[0159] The server distributes the analysis results and sentiment reports to all participants, helping them understand areas for improvement for the next meeting.

[0160] Step 8:

[0161] Users review the reports sent from the server and use the meeting's outcomes and improvement suggestions to prepare for the next meeting.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] In today's business environment, there is a demand for improved meeting efficiency and productivity. In particular, insufficient understanding of participants' emotions regarding discussions can lead to a decline in the quality of decision-making and a decrease in the effectiveness of meetings. Furthermore, the lack of means to analyze emotions and provide feedback in real time makes it difficult to manage meetings flexibly according to the situation.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes means for receiving audio signals and analyzing language data and emotional states; means for evaluating participants' responses based on the analyzed emotional states and adjusting the progress of the meeting; and means for aggregating emotional analysis data and generating a report for evaluating the quality of the meeting. This makes it possible to accurately grasp the emotional shifts of participants during and after the meeting, and to provide feedback and adjust the progress of the meeting based on that.

[0167] "Participant schedule information" refers to data about the time available to meeting attendees, including their free time and existing appointments.

[0168] The "method for suggesting meeting times" is a function that calculates and presents the optimal meeting start time based on the participants' schedule information.

[0169] A "meeting schedule" is a plan that outlines the agenda and the order in which the meeting will proceed, based on its objectives.

[0170] An "audio signal" is data obtained by electrically converting sound received through a microphone or similar device.

[0171] "Language data" refers to information about characters and phrases extracted from speech signals using speech recognition technology.

[0172] "Emotional state" refers to information about an individual's emotional response, obtained through an analysis of their tone of voice and word choice.

[0173] "Means of adjusting the progress" refers to a function that dynamically adjusts the pace of discussion and topics based on information gathered during a meeting.

[0174] "Emotional analysis data" refers to numerical data representing statistics and trends related to emotions, analyzed based on participants' statements.

[0175] The "means of generating reports" refer to a function that uses data accumulated after a meeting to create a report summarizing the content of the meeting and the results of an analysis of the emotions expressed.

[0176] This invention provides a system that enhances the quality of meeting management and precisely analyzes the emotional state of participants. The system consists of a server and terminals, with the central server integrating speech recognition technology and emotion analysis technology. The following describes how this system is implemented.

[0177] The server uses commercially available speech recognition software as its speech recognition engine. Examples of such software include Google Cloud Speech-to-Text and Amazon Transcribe. The server utilizes this software to receive audio signals from meeting participants in real time and convert them into language data.

[0178] Furthermore, Microsoft Azure Text Analytics and IBM Watson® Tone Analyzer can be used as sentiment analysis engines. By using these, it is possible to infer participants' emotional states from language data. The server collects the obtained sentiment data, evaluates participants' emotional trends in real time, and adjusts the progress accordingly.

[0179] For example, the server uses sentiment analysis to determine that "Participant A is showing a negative reaction to the presentation" and sends a notification to the device. The device then provides feedback to the user, such as prompting them to move on to the next agenda item.

[0180] In addition, after the meeting ends, the server integrates all the collected data and generates a report summarizing the sentiment analysis results for the entire meeting. For example, by entering the prompt "Analyze the sentiments of meeting participants regarding the latest product presentation and summarize their impressions," an analysis based on the generated AI model will be performed.

[0181] This allows users to understand participants' emotions and consider reviewing and improving the content of future meetings. Through this system, improvements in meeting productivity and participant satisfaction are expected.

[0182] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0183] Step 1:

[0184] The server receives audio signals from meeting participants. These audio signals are acquired through microphones and other means and are first converted into digital data. The input is an analog audio signal, and the output is digital audio data. This makes it possible to process the audio using a speech recognition engine.

[0185] Step 2:

[0186] The server sends digital audio data to a speech recognition process, which converts it to text. Speech recognition software is used here, for example, Google Cloud Speech-to-Text. The input is digital audio data, and the output is real-time text data. Specifically, a person's speech is transcribed sequentially by the computer.

[0187] Step 3:

[0188] The server sends the converted text data to the sentiment analysis engine. Using tools such as Microsoft Azure Text Analytics, the engine analyzes the text data to determine the emotional state. The input is text data, and the output is data indicating the emotional state. At this stage, specific keywords and sentence tones are analyzed to identify the participant's emotions at that time.

[0189] Step 4:

[0190] Based on the analysis results, the server sends meeting progress adjustment suggestions to the terminal. It evaluates the sentiment analysis data and generates feedback that takes into account the participants' level of interest and emotional trends. The input is emotional state data, and the output is information related to progress adjustments. Specifically, a notification such as "Negative reactions to agenda item A are increasing" is conveyed to the user via the terminal.

[0191] Step 5:

[0192] After the meeting concludes, the server summarizes all the analysis data and compiles the meeting's sentiment trends into a final report. The report is automatically generated based on prompts generated by a generative AI model. The input is the accumulated sentiment analysis data, and the output is a meeting evaluation report. This report provides users with concrete information to help them prepare for and strategize for future meetings.

[0193] (Application Example 2)

[0194] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0195] Traditional meeting systems were inefficient because meeting progress management and agenda generation were done manually. Furthermore, the lack of real-time monitoring of participants' comments and emotional states meant that meeting quality was easily influenced by participants' subjective opinions. In addition, in service industries, understanding customers' emotional states and improving responses accordingly was difficult. This hindered improvements in meeting productivity and customer satisfaction.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0197] In this invention, the server includes means for acquiring time management information of participants and proposing the optimal meeting time, means for analyzing emotional states during conversations and notifying participants of their level of interest and emotional responses in real time, and means used during conversations with customers to support improvements in customer service based on the customer's emotional state. This enables not only increased efficiency and improved quality of meetings, but also the optimization of customer service in physical stores.

[0198] "Time management information" refers to data that includes participants' schedules and appointments, and is used to optimize the timing of meetings and other activities.

[0199] An "agenda" refers to a plan that lists the topics or themes to be discussed in a meeting or discussion.

[0200] "Documenting speech" is the process of converting spoken words into text for recording, and it is important for later review and analysis of what was said in a meeting.

[0201] A "task list" is a list of specific tasks or activities that need to be completed, and it is notified individually to the participants or those responsible.

[0202] "Emotional state" refers to individual psychological reactions and feelings that can be interpreted from voice and facial expressions, and is classified through emotional analysis.

[0203] "Customer service improvement" refers to initiatives and processes aimed at improving the quality of service and interaction with customers.

[0204] The system for implementing this invention processes data in real time by combining speech recognition and sentiment analysis technologies. The server receives audio data from the microphone of a smartphone or smart glasses. This audio data is converted to text using a speech recognition engine in the cloud (e.g., Google Cloud Speech-to-Text API). Next, sentiment analysis is performed on this text data using a natural language processing engine (e.g., IBM Watson Natural Language Understanding).

[0205] Furthermore, the server sends real-time notifications to users based on the analysis results. These notifications provide feedback tailored to the participants' level of interest and emotions, helping to improve meeting flow and customer service. Specifically, if a customer shows interest, the user can make suggestions that emphasize that point.

[0206] A concrete example is when a system detects that a customer is showing interest in a new product and has a positive emotional response to it while it's being introduced in a physical store. Based on this information, the system can then provide the user with more detailed information about the product or encourage them to purchase it.

[0207] An example of a prompt for a generative AI model is: "During a conversation with a customer, analyze the customer's emotions from the following audio data. Determine whether they are positive, negative, or neutral and notify us in real time."

[0208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0209] Step 1:

[0210] The device uses a microphone to collect ambient sound in real time. The audio data is recorded as a digital signal and sent to the server as input data.

[0211] Step 2:

[0212] The server sends the received audio data to the speech recognition engine. This engine uses the Google Cloud Speech-to-Text API to convert the audio into text data. Data processing is performed to make the audio signal a format that can be processed by language. The converted text becomes the input for the next step.

[0213] Step 3:

[0214] The server sends the text data to the sentiment analysis engine. This engine uses IBM Watson Natural Language Understanding to extract emotional states from the text. Specifically, it analyzes keywords and tone within the text to determine emotional categories such as positive, negative, and neutral. The resulting emotional data is the output for the next step.

[0215] Step 4:

[0216] The server generates appropriate feedback based on the analysis results. It sends prompts to the generating AI model to create feedback messages that match the situation. These prompts are based on the analyzed sentiment data and may include content such as, "The customer appears interested in the product."

[0217] Step 5:

[0218] The device notifies the user of feedback messages. These notifications enable the user to take appropriate actions and improvements in real time, depending on the situation. Prompt messages are displayed visually or audibly through information dissemination tools to inform the user.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] The meeting support system according to the present invention is designed to embody a series of processes for efficiently operating and managing meetings throughout the pre-meeting preparation, during the meeting, and after the meeting. Its embodiments are described in detail below.

[0236] The core of this system lies in a server, which collects data from the APIs of the scheduling management systems used by each participant to obtain their schedule information. The server analyzes the collected data and automatically suggests the optimal meeting time by comparing the availability of all participants.

[0237] The server then automatically generates an agenda based on the meeting's objectives. It utilizes past meeting records and purpose-specific templates to set appropriate topics. This generated agenda is then communicated to participants via email or a dedicated application, allowing them to review it before the meeting.

[0238] During the meeting, the server uses speech recognition technology to transcribe participants' statements in real time. The server analyzes the audio data, identifies speakers, and automatically generates meeting minutes. This feature allows participants to focus on the discussion and reduces the effort required for record-keeping.

[0239] After the meeting, the server automatically generates meeting minutes, summarizes them, extracts key points, and distributes them to participants. It also organizes the action items decided during the meeting and notifies the responsible parties with specific deadlines. During this process, the terminal displays a to-do list to the user, allowing them to manage their progress.

[0240] Furthermore, when scheduling the next meeting, the server collects feedback from participants and, based on that, proposes the most suitable dates and times again. Iterating through this process promotes further efficiency.

[0241] The data from meetings is analyzed in detail by a server, recording things like the frequency of each speaker and the flow of topics during the meeting. Based on this data, points for improvement in future meetings are suggested. These suggestions are distributed to each participant as a dedicated report, serving as foundational material for improving the quality of meetings.

[0242] For example, if the server sets the meeting theme to "Market launch of a new product," it will automatically generate an agenda aligned with that theme and send it to the participants. As the meeting progresses, the server will record each statement and guide the participants to efficiently carry out the next steps.

[0243] In this way, servers, terminals, and users work together within the entire system, enabling efficient meeting management. In particular, the server's data analysis and optimization algorithms contribute to simplifying meeting preparation and reducing the burden on participants.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The server retrieves schedule information from participants' schedule management systems via API. All participants' schedules are stored in a database and prepared for analysis.

[0247] Step 2:

[0248] The server compares each participant's schedule and extracts common available time slots. This allows it to list the optimal meeting times that all participants can attend.

[0249] Step 3:

[0250] The server automatically generates an agenda based on the meeting's objectives, using past meeting records and preset templates. The generated agenda is then sent to all participants via email or a dedicated app.

[0251] Step 4:

[0252] When a meeting begins, the terminal sends the meeting audio to the server. The server uses a speech recognition engine to transcribe the audio into text in real time, creating the basic data for the meeting minutes.

[0253] Step 5:

[0254] The server analyzes the transcribed data, organizes the content for each speaker, and incorporates it into the meeting minutes. Simultaneously, it monitors the progress of the meeting and manages time according to the agenda. It issues instructions to move on to the next topic as needed.

[0255] Step 6:

[0256] After the meeting ends, the server automatically summarizes the generated minutes and extracts key decisions and to-do lists. Based on this, it sends a report to participants via email.

[0257] Step 7:

[0258] The server organizes the action items decided during the meeting by person in charge, sets specific deadlines, and notifies each participant via their terminal as a to-do list.

[0259] Step 8:

[0260] The server adjusts the next meeting date based on feedback collected from participants. It presents participants with proposed dates and times to facilitate the scheduling process.

[0261] Step 9:

[0262] The server analyzes the data collected during the meeting to evaluate the frequency of participation and the progress of the discussion. Based on the analysis results, improvement suggestions are compiled into a report and provided to the participants.

[0263] (Example 1)

[0264] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] In today's business environment, efficient meeting management is a critical challenge. Especially when many participants are involved, significant effort is required to optimize meeting times, create agendas, and record and share meeting content. Furthermore, it's essential to properly manage the action items decided upon after the meeting and use them to improve future meetings. Traditional systems fail to efficiently address these challenges, making it difficult to improve the quality of meetings.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0267] In this invention, the server includes means for acquiring participants' schedule information and suggesting the optimal time, means for automatically generating an agenda based on the purpose of the meeting and distributing it to participants, and means for analyzing speech and transcribing speech into text in real time. As a result, meeting preparation time is significantly reduced, participants can concentrate on the discussion, post-meeting action item management is streamlined, and the overall quality of the meeting can be improved.

[0268] "Participant schedule information" refers to data on the schedules and available times of individuals participating in the meeting.

[0269] "Suggesting the optimal time" is the process of selecting the most appropriate and efficient meeting start time, taking into account the schedules of all participants.

[0270] A "meeting agenda based on the purpose of the meeting" is a list of topics or subjects set in line with the focus and goals to be achieved at the meeting.

[0271] "Automatic generation" refers to a system creating content based on predetermined algorithms and data, without manual human intervention.

[0272] "Analyzing speech and converting it to text in real time" refers to the process of instantly converting spoken words during a meeting into text information using speech recognition technology.

[0273] "Automatically generating records" means that the system independently organizes the collected data and creates records based on that data.

[0274] A "list of individual tasks" is a list that summarizes the tasks and responsibilities decided during the meeting for each participant.

[0275] "Automatically adjusting the next meeting date" is a process that re-examines the availability of meeting participants and suggests the optimal date and time for the next meeting.

[0276] "Analyzing data and making improvement suggestions based on participants' comments" means analyzing recorded comment data, identifying areas for improvement in meeting management methods, and proposing solutions.

[0277] "Generating a document describing technical features" involves creating a document that summarizes the meeting content from a technical perspective, based on meeting records and key points.

[0278] This conference support system is designed to facilitate efficient communication between servers, terminals, and users. Specific implementations using hardware and software are described below.

[0279] hardware

[0280] The server retrieves participants' schedule information using the APIs of each scheduling management system. Specifically, it collects each participant's available time using the Google Calendar API and the Microsoft Graph API. The server uses a high-performance server computing unit suitable for processing large amounts of data.

[0281] software

[0282] The system software running on the server analyzes the acquired schedule data and executes an algorithm to calculate the optimal meeting time. For speech recognition technology, Azure Speech Service and Google Speech-to-Text API are used. This technology transcribes speech during meetings into text in real time and automatically generates meeting minutes based on the content of the speech.

[0283] The server automatically generates an agenda based on the purpose of the meeting using a generation AI model. For this, a template-based generation AI model is utilized, and by inputting a prompt sentence such as "Please create agenda items regarding the market introduction of a new product", the AI proposes appropriate agenda items.

[0284] Usage example

[0285] Users can access from a terminal and check the details of the meeting through the platform. For example, the candidate date and time for the next meeting proposed by the server are generated based on the user's feedback. With a prompt sentence like "Please tell me the optimal meeting time considering the schedules of all participants", the server proposes the optimal candidates by AI.

[0286] With this system, a series of processes from meeting preparation to implementation and adjustment of the next meeting are streamlined, and users can more easily concentrate on the discussion. The data generated between each process is continuously analyzed by the server, contributing to the improvement of the quality of meetings in subsequent sessions. In this way, it is possible to achieve efficiency and quality improvement throughout the entire meeting.

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The server obtains schedule information from the API of the schedule management system used by each participant. This input data is information regarding the schedules of individual participants. The server collects this and saves it in a database to list up the free time of all participants. As data processing, an algorithm for removing overlapping times and non-business hours is executed.

[0290] Step 2:

[0291] The server compares and analyzes the availability of all participants based on the stored schedule data. The input data is the schedule information processed in Step 1. The server calculates and proposes the optimal meeting time. In this process, a generative AI model is used, and prompts such as "Please tell me the optimal meeting time considering the availability of all participants" are used to have the AI ​​calculate the recommended time. The optimal meeting time is generated as output.

[0292] Step 3:

[0293] The server receives the meeting objective and automatically generates a corresponding agenda using an AI model. The input at this stage is information about the meeting objective. It selects and constructs appropriate topics and subjects using past meeting records and templates. The output is the automatically generated agenda distributed to participants. A specific example of its operation is using the prompt, "Please create agenda items regarding the market launch of the new product."

[0294] Step 4:

[0295] During the meeting, the server captures audio in real time and uses speech recognition technology to transcribe the speech into text. The input data is the audio data collected during the meeting. The audio data is analyzed, separated by speaker, and output as text data. The system operates by using a speech recognition API (e.g., Azure Speech Service) to perform simultaneous text transcription.

[0296] Step 5:

[0297] The server automatically generates meeting minutes based on text data obtained through speech recognition. The input data is the text data generated in step 4. The server summarizes this data, extracts key points, and structures it as meeting minutes. The output is the meeting minutes, which are shared after the meeting. The server has the function to distribute the meeting minutes electronically to participants.

[0298] Step 6:

[0299] The server organizes the action items decided during the meeting and creates a task list for each person in charge. This uses the meeting minutes created in step 5 as input. The action items are notified along with the person in charge and the deadline, and the output is an individual task list. The terminal displays this task list to the user, allowing them to track and check the progress.

[0300] Step 7:

[0301] The server collects feedback from participants to optimize the date and time of the next meeting. The input data is feedback collected from participants. Based on this, a generative AI model and the prompt "Please tell me the best date and time for the next meeting" are used to calculate and suggest candidate dates and times. The output is the adjusted candidate date and time for the next meeting.

[0302] Step 8:

[0303] The server meticulously analyzes the speech data collected during and after the meeting. The input data consists of speech data from all meeting participants. The server analyzes the frequency of speech and the flow of discussion to identify areas for improvement. The output is provided to participants as a report to help improve the quality of future meetings. A data analysis algorithm is used in its operation.

[0304] (Application Example 1)

[0305] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0306] While efficiency improvements are needed in data center and other facility operations, such as meetings and maintenance, coordinating participant schedules, creating agendas, and managing meeting minutes still require considerable time and effort. Furthermore, facility maintenance and procurement of necessary parts are predominantly managed manually, often leading to inefficiencies. In this context, there is a need to achieve efficiency and automation across the entire process of meeting management and facility management.

[0307] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following respective means.

[0308] In this invention, the server includes means for acquiring the schedule information of participants and proposing an optimal meeting time, means for automatically generating items of a meeting based on the purpose and communicating them to the participants, means for analyzing the voice during the meeting and converting the speech into text in real time, means for automatically generating a record based on the speech content and sharing it with the participants after the meeting, means for coordinating the status of facilities and the maintenance activity schedule and proposing an optimal maintenance activity time, and means for automatically placing orders for necessary equipment and parts. Thereby, the burden of meeting operation is reduced and efficient facility management becomes possible.

[0309] "Participants" are people who are involved in meetings or activities and have the role of providing schedules and opinions.

[0310] "Meeting time" refers to the time when participants gather for information exchange and decision-making.

[0311] "Agenda" is a list of topics and contents to be handled in the progress of a meeting and indicates the purpose and direction of the meeting.

[0312] "Voice analysis" refers to a technology that acquires spoken words as voice data and converts them into character information.

[0313] "Transcription of speech" is a process of recording the content of speech acquired as voice as digital characters.

[0314] "Record" is a document or data that stores the content discussed during a meeting in a form that can be referred to later.

[0315] "Maintenance activity" refers to regular maintenance and repair activities performed to maintain the normal operation of facilities and systems.

[0316] "Equipment and parts" refers to the physical elements and replaceable parts necessary for the operation of the equipment.

[0317] "Automated ordering" is a process that orders necessary goods based on predetermined conditions, with minimal human involvement.

[0318] This system is built around a multi-functional server. The server utilizes cloud services, such as AWS Lambda and DynamoDB, to efficiently process data. The server can retrieve participants' schedule information from the Google Calendar API, analyze it, and automatically suggest the optimal meeting time. This information is then distributed to participants via smartphones and computers.

[0319] Furthermore, the server uses generative AI models such as OpenAI's GPT-3 to generate an agenda based on the purpose of the meeting. This makes it possible to notify participants of the meeting's flow in advance and encourage them to prepare. During the meeting, Google Cloud Speech-to-Text is used to transcribe speeches in real time, and this data is stored in DynamoDB.

[0320] Furthermore, after the meeting, the server summarizes key points based on accumulated data and individually notifies participants of specific action items. The server also calculates the optimal timing for maintenance activities based on the equipment status and maintenance schedule, and automatically places orders for necessary equipment and parts in advance.

[0321] For example, in a maintenance meeting for cooling systems within a data center, the server can suggest the optimal repair time and place orders considering parts inventory. A possible prompt request might be, "Please suggest the optimal date for the next cooling system repair, taking into account the team members' schedules."

[0322] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0323] Step 1:

[0324] The server retrieves participants' schedule information via the Google Calendar API. The input is each participant's calendar access permission, and the server collects their schedule data based on this. The output is schedule data including the free time slots of all participants. Based on this, the server calculates the intersection of free time slots and proposes the optimal meeting time.

[0325] Step 2:

[0326] The server uses the collected data and leverages OpenAI's GPT-3 generative AI model to automatically generate an agenda based on the meeting's objectives. Inputs include the meeting's theme and past meeting minutes. Output is a detailed meeting agenda sent to participants. The server distributes this agenda to devices via email or a dedicated application.

[0327] Step 3:

[0328] During the meeting, the server uses Google Cloud Speech-to-Text to transcribe participants' speech in real time. The input is audio data collected during the meeting, and the output is transcribed text data. The server stores this data in DynamoDB for later processing.

[0329] Step 4:

[0330] After the meeting concludes, the server analyzes the accumulated text data and summarizes the key points. The input is the transcribed and saved meeting minutes, and the output is the summarized meeting minutes. The server uses this summary to send each participant a to-do list of action items.

[0331] Step 5:

[0332] The server analyzes equipment status and maintenance logs to plan the optimal timing for maintenance activities. Inputs include equipment operating status data and repair history data. Outputs are specific activity schedules notified to the maintenance team. Furthermore, the server predicts necessary equipment and parts and places purchase orders through an automated ordering system.

[0333] Step 6:

[0334] The user uses prompts to send details about the next meeting and other specific requests to the generating AI model. The input is a text-based request from the user (e.g., "Please suggest the best date for the next cooling system repair, taking into account the team members' schedules"). The output is the optimal suggestion generated by the AI ​​and displayed on the user's device.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] This invention is a system designed to improve the quality of meetings by not only streamlining meeting management but also analyzing the emotional state of participants. This system utilizes artificial intelligence, combining speech recognition and emotion analysis to grasp participants' emotions in real time and provide feedback to facilitate the meeting.

[0337] The central server of the system receives audio data from meeting participants and first uses a speech recognition engine to transcribe their speech into text in real time. Simultaneously, an emotion engine analyzes speech patterns and linguistic expressions to detect emotional states. For example, it can identify positive, negative, or neutral emotions based on the tone and speed of speech, as well as specific keywords.

[0338] By performing sentiment analysis in real time, the server sends instructions to the terminal to appropriately adjust the progress of the meeting. The terminal can then notify the user, for example, "Participant A may be losing interest" or "The agenda item seems to have suddenly elicited a negative reaction," prompting adjustments to the ongoing agenda.

[0339] After the meeting, the server analyzes the data obtained by the emotion engine and generates a detailed report summarizing emotional trends and responses to specific topics, which is then distributed to participants. This provides material for concretely considering improvements for the next meeting and supports better decision-making.

[0340] For example, if participants' emotions regarding a particular product topic show a generally negative trend during a meeting, the emotional data will be used to consider areas for product improvement or switching to a different topic. This feedback will be provided after the meeting and used to prepare for the next meeting.

[0341] This system enables the server to manage meetings dynamically based on emotional data, and makes it easier for users to grasp the overall atmosphere of the meeting. This increases the likelihood of improved satisfaction for all participants and increased meeting productivity.

[0342] The following describes the processing flow.

[0343] Step 1:

[0344] The server accesses each participant's calendar API to retrieve their schedule information and collects their appointments. Based on this, it identifies the optimal meeting time and proposes it to all participants.

[0345] Step 2:

[0346] The server automatically generates an agenda based on the meeting's objectives. The generated agenda is then distributed to participants via email or a dedicated application.

[0347] Step 3:

[0348] Once the meeting begins, the terminal sends the audio data from the meeting to the server. The server uses a speech recognition engine to transcribe the spoken words into text in real time.

[0349] Step 4:

[0350] The server has an emotion engine built in that analyzes audio data to identify the emotional state of participants. It assigns positive, negative, and neutral emotion tags.

[0351] Step 5:

[0352] Based on the results of sentiment analysis, the server evaluates the progress of the meeting in real time and suggests adjusting the agenda to the user via the terminal at the appropriate time.

[0353] Step 6:

[0354] After the meeting ends, the server generates meeting minutes and comprehensively analyzes participants' reactions based on sentiment data. This then generates a detailed sentiment report.

[0355] Step 7:

[0356] The server distributes the analysis results and sentiment reports to all participants, helping them understand areas for improvement for the next meeting.

[0357] Step 8:

[0358] Users review the reports sent from the server and use the meeting's outcomes and improvement suggestions to prepare for the next meeting.

[0359] (Example 2)

[0360] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0361] In today's business environment, there is a demand for improved meeting efficiency and productivity. In particular, insufficient understanding of participants' emotions regarding discussions can lead to a decline in the quality of decision-making and a decrease in the effectiveness of meetings. Furthermore, the lack of means to analyze emotions and provide feedback in real time makes it difficult to manage meetings flexibly according to the situation.

[0362] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0363] In this invention, the server includes means for receiving audio signals and analyzing language data and emotional states; means for evaluating participants' responses based on the analyzed emotional states and adjusting the progress of the meeting; and means for aggregating emotional analysis data and generating a report for evaluating the quality of the meeting. This makes it possible to accurately grasp the emotional shifts of participants during and after the meeting, and to provide feedback and adjust the progress of the meeting based on that.

[0364] "Participant schedule information" refers to data about the time available to meeting attendees, including their free time and existing appointments.

[0365] The "method for suggesting meeting times" is a function that calculates and presents the optimal meeting start time based on the participants' schedule information.

[0366] A "meeting schedule" is a plan that outlines the agenda and the order in which the meeting will proceed, based on its objectives.

[0367] An "audio signal" is data obtained by electrically converting sound received through a microphone or similar device.

[0368] "Language data" refers to information about characters and phrases extracted from speech signals using speech recognition technology.

[0369] "Emotional state" refers to information about an individual's emotional response, obtained through an analysis of their tone of voice and word choice.

[0370] "Means of adjusting the progress" refers to a function that dynamically adjusts the pace of discussion and topics based on information gathered during a meeting.

[0371] "Emotional analysis data" refers to numerical data representing statistics and trends related to emotions, analyzed based on participants' statements.

[0372] The "means of generating reports" refer to a function that uses data accumulated after a meeting to create a report summarizing the content of the meeting and the results of an analysis of the emotions expressed.

[0373] This invention provides a system that enhances the quality of meeting management and precisely analyzes the emotional state of participants. The system consists of a server and terminals, with the central server integrating speech recognition technology and emotion analysis technology. The following describes how this system is implemented.

[0374] The server uses commercially available speech recognition software as its speech recognition engine. Examples of such software include Google Cloud Speech-to-Text and Amazon Transcribe. The server utilizes this software to receive audio signals from meeting participants in real time and convert them into language data.

[0375] Furthermore, Microsoft Azure Text Analytics and IBM Watson Tone Analyzer can be used as sentiment analysis engines. These allow for the inference of participants' emotional states from linguistic data. The server collects the acquired sentiment data, evaluates participants' emotional trends in real time, and adjusts the program accordingly.

[0376] For example, the server uses sentiment analysis to determine that "Participant A is showing a negative reaction to the presentation" and sends a notification to the device. The device then provides feedback to the user, such as prompting them to move on to the next agenda item.

[0377] In addition, after the meeting ends, the server integrates all the collected data and generates a report summarizing the sentiment analysis results for the entire meeting. For example, by entering the prompt "Analyze the sentiments of meeting participants regarding the latest product presentation and summarize their impressions," an analysis based on the generated AI model will be performed.

[0378] This allows users to understand participants' emotions and consider reviewing and improving the content of future meetings. Through this system, improvements in meeting productivity and participant satisfaction are expected.

[0379] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0380] Step 1:

[0381] The server receives audio signals from meeting participants. These audio signals are acquired through microphones and other means and are first converted into digital data. The input is an analog audio signal, and the output is digital audio data. This makes it possible to process the audio using a speech recognition engine.

[0382] Step 2:

[0383] The server sends digital audio data to a speech recognition process, which converts it to text. Speech recognition software is used here, for example, Google Cloud Speech-to-Text. The input is digital audio data, and the output is real-time text data. Specifically, a person's speech is transcribed sequentially by the computer.

[0384] Step 3:

[0385] The server sends the converted text data to the sentiment analysis engine. Using tools such as Microsoft Azure Text Analytics, the engine analyzes the text data to determine the emotional state. The input is text data, and the output is data indicating the emotional state. At this stage, specific keywords and sentence tones are analyzed to identify the participant's emotions at that time.

[0386] Step 4:

[0387] Based on the analysis results, the server sends meeting progress adjustment suggestions to the terminal. It evaluates the sentiment analysis data and generates feedback that takes into account the participants' level of interest and emotional trends. The input is emotional state data, and the output is information related to progress adjustments. Specifically, a notification such as "Negative reactions to agenda item A are increasing" is conveyed to the user via the terminal.

[0388] Step 5:

[0389] After the meeting concludes, the server summarizes all the analysis data and compiles the meeting's sentiment trends into a final report. The report is automatically generated based on prompts generated by a generative AI model. The input is the accumulated sentiment analysis data, and the output is a meeting evaluation report. This report provides users with concrete information to help them prepare for and strategize for future meetings.

[0390] (Application Example 2)

[0391] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0392] Traditional meeting systems were inefficient because meeting progress management and agenda generation were done manually. Furthermore, the lack of real-time monitoring of participants' comments and emotional states meant that meeting quality was easily influenced by participants' subjective opinions. In addition, in service industries, understanding customers' emotional states and improving responses accordingly was difficult. This hindered improvements in meeting productivity and customer satisfaction.

[0393] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0394] In this invention, the server includes means for acquiring time management information of participants and proposing the optimal meeting time, means for analyzing emotional states during conversations and notifying participants of their level of interest and emotional responses in real time, and means used during conversations with customers to support improvements in customer service based on the customer's emotional state. This enables not only increased efficiency and improved quality of meetings, but also the optimization of customer service in physical stores.

[0395] "Time management information" refers to data that includes participants' schedules and appointments, and is used to optimize the timing of meetings and other activities.

[0396] An "agenda" refers to a plan that lists the topics or themes to be discussed in a meeting or discussion.

[0397] "Documenting speech" is the process of converting spoken words into text for recording, and it is important for later review and analysis of what was said in a meeting.

[0398] A "task list" is a list of specific tasks or activities that need to be completed, and it is notified individually to the participants or those responsible.

[0399] "Emotional state" refers to individual psychological reactions and feelings that can be interpreted from voice and facial expressions, and is classified through emotional analysis.

[0400] "Customer service improvement" refers to initiatives and processes aimed at improving the quality of service and interaction with customers.

[0401] The system for implementing this invention processes data in real time by combining speech recognition and sentiment analysis technologies. The server receives audio data from the microphone of a smartphone or smart glasses. This audio data is converted to text using a speech recognition engine in the cloud (e.g., Google Cloud Speech-to-Text API). Next, sentiment analysis is performed on this text data using a natural language processing engine (e.g., IBM Watson Natural Language Understanding).

[0402] Furthermore, the server sends real-time notifications to users based on the analysis results. These notifications provide feedback tailored to the participants' level of interest and emotions, helping to improve meeting flow and customer service. Specifically, if a customer shows interest, the user can make suggestions that emphasize that point.

[0403] A concrete example is when a system detects that a customer is showing interest in a new product and has a positive emotional response to it while it's being introduced in a physical store. Based on this information, the system can then provide the user with more detailed information about the product or encourage them to purchase it.

[0404] An example of a prompt for a generative AI model is: "During a conversation with a customer, analyze the customer's emotions from the following audio data. Determine whether they are positive, negative, or neutral and notify us in real time."

[0405] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0406] Step 1:

[0407] The device uses a microphone to collect ambient sound in real time. The audio data is recorded as a digital signal and sent to the server as input data.

[0408] Step 2:

[0409] The server sends the received audio data to the speech recognition engine. This engine uses the Google Cloud Speech-to-Text API to convert the audio into text data. Data processing is performed to make the audio signal a format that can be processed by language. The converted text becomes the input for the next step.

[0410] Step 3:

[0411] The server sends the text data to the sentiment analysis engine. This engine uses IBM Watson Natural Language Understanding to extract emotional states from the text. Specifically, it analyzes keywords and tone within the text to determine emotional categories such as positive, negative, and neutral. The resulting emotional data is the output for the next step.

[0412] Step 4:

[0413] The server generates appropriate feedback based on the analysis results. It sends prompts to the generating AI model to create feedback messages that match the situation. These prompts are based on the analyzed sentiment data and may include content such as, "The customer appears interested in the product."

[0414] Step 5:

[0415] The device notifies the user of feedback messages. These notifications enable the user to take appropriate actions and improvements in real time, depending on the situation. Prompt messages are displayed visually or audibly through information dissemination tools to inform the user.

[0416] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0417] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0418] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0419] [Third Embodiment]

[0420] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0421] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0422] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0423] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0424] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0425] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0426] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0427] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0428] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0429] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0430] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0431] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0432] The meeting support system according to the present invention is designed to embody a series of processes for efficiently operating and managing meetings throughout the pre-meeting preparation, during the meeting, and after the meeting. Its embodiments are described in detail below.

[0433] The core of this system lies in a server, which collects data from the APIs of the scheduling management systems used by each participant to obtain their schedule information. The server analyzes the collected data and automatically suggests the optimal meeting time by comparing the availability of all participants.

[0434] The server then automatically generates an agenda based on the meeting's objectives. It utilizes past meeting records and purpose-specific templates to set appropriate topics. This generated agenda is then communicated to participants via email or a dedicated application, allowing them to review it before the meeting.

[0435] During the meeting, the server uses speech recognition technology to transcribe participants' statements in real time. The server analyzes the audio data, identifies speakers, and automatically generates meeting minutes. This feature allows participants to focus on the discussion and reduces the effort required for record-keeping.

[0436] After the meeting, the server automatically generates meeting minutes, summarizes them, extracts key points, and distributes them to participants. It also organizes the action items decided during the meeting and notifies the responsible parties with specific deadlines. During this process, the terminal displays a to-do list to the user, allowing them to manage their progress.

[0437] Furthermore, when scheduling the next meeting, the server collects feedback from participants and, based on that, proposes the most suitable dates and times again. Iterating through this process promotes further efficiency.

[0438] The data from meetings is analyzed in detail by a server, recording things like the frequency of each speaker and the flow of topics during the meeting. Based on this data, points for improvement in future meetings are suggested. These suggestions are distributed to each participant as a dedicated report, serving as foundational material for improving the quality of meetings.

[0439] For example, if the server sets the meeting theme to "Market launch of a new product," it will automatically generate an agenda aligned with that theme and send it to the participants. As the meeting progresses, the server will record each statement and guide the participants to efficiently carry out the next steps.

[0440] In this way, servers, terminals, and users work together within the entire system, enabling efficient meeting management. In particular, the server's data analysis and optimization algorithms contribute to simplifying meeting preparation and reducing the burden on participants.

[0441] The following describes the processing flow.

[0442] Step 1:

[0443] The server retrieves schedule information from participants' schedule management systems via API. All participants' schedules are stored in a database and prepared for analysis.

[0444] Step 2:

[0445] The server compares each participant's schedule and extracts common available time slots. This allows it to list the optimal meeting times that all participants can attend.

[0446] Step 3:

[0447] The server automatically generates an agenda based on the meeting's objectives, using past meeting records and preset templates. The generated agenda is then sent to all participants via email or a dedicated app.

[0448] Step 4:

[0449] When a meeting begins, the terminal sends the meeting audio to the server. The server uses a speech recognition engine to transcribe the audio into text in real time, creating the basic data for the meeting minutes.

[0450] Step 5:

[0451] The server analyzes the transcribed data, organizes the content for each speaker, and incorporates it into the meeting minutes. Simultaneously, it monitors the progress of the meeting and manages time according to the agenda. It issues instructions to move on to the next topic as needed.

[0452] Step 6:

[0453] After the meeting ends, the server automatically summarizes the generated minutes and extracts key decisions and to-do lists. Based on this, it sends a report to participants via email.

[0454] Step 7:

[0455] The server organizes the action items decided during the meeting by person in charge, sets specific deadlines, and notifies each participant via their terminal as a to-do list.

[0456] Step 8:

[0457] The server adjusts the next meeting date based on feedback collected from participants. It presents participants with proposed dates and times to facilitate the scheduling process.

[0458] Step 9:

[0459] The server analyzes the data collected during the meeting to evaluate the frequency of participation and the progress of the discussion. Based on the analysis results, improvement suggestions are compiled into a report and provided to the participants.

[0460] (Example 1)

[0461] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] In today's business environment, efficient meeting management is a critical challenge. Especially when many participants are involved, significant effort is required to optimize meeting times, create agendas, and record and share meeting content. Furthermore, it's essential to properly manage the action items decided upon after the meeting and use them to improve future meetings. Traditional systems fail to efficiently address these challenges, making it difficult to improve the quality of meetings.

[0463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0464] In this invention, the server includes means for acquiring participants' schedule information and suggesting the optimal time, means for automatically generating an agenda based on the purpose of the meeting and distributing it to participants, and means for analyzing speech and transcribing speech into text in real time. As a result, meeting preparation time is significantly reduced, participants can concentrate on the discussion, post-meeting action item management is streamlined, and the overall quality of the meeting can be improved.

[0465] "Participant schedule information" refers to data on the schedules and available times of individuals participating in the meeting.

[0466] "Suggesting the optimal time" is the process of selecting the most appropriate and efficient meeting start time, taking into account the schedules of all participants.

[0467] A "meeting agenda based on the purpose of the meeting" is a list of topics or subjects set in line with the focus and goals to be achieved at the meeting.

[0468] "Automatic generation" refers to a system creating content based on predetermined algorithms and data, without manual human intervention.

[0469] "Analyzing speech and converting it to text in real time" refers to the process of instantly converting spoken words during a meeting into text information using speech recognition technology.

[0470] "Automatically generating records" means that the system independently organizes the collected data and creates records based on that data.

[0471] A "list of individual tasks" is a list that summarizes the tasks and responsibilities decided during the meeting for each participant.

[0472] "Automatically adjusting the next meeting date" is a process that re-examines the availability of meeting participants and suggests the optimal date and time for the next meeting.

[0473] "Analyzing data and making improvement suggestions based on participants' comments" means analyzing recorded comment data, identifying areas for improvement in meeting management methods, and proposing solutions.

[0474] "Generating a document describing technical features" involves creating a document that summarizes the meeting content from a technical perspective, based on meeting records and key points.

[0475] This conference support system is designed to facilitate efficient communication between servers, terminals, and users. Specific implementations using hardware and software are described below.

[0476] hardware

[0477] The server retrieves participants' schedule information using the APIs of each scheduling management system. Specifically, it collects each participant's available time using the Google Calendar API and the Microsoft Graph API. The server uses a high-performance server computing unit suitable for processing large amounts of data.

[0478] software

[0479] The system software running on the server analyzes the acquired schedule data and executes an algorithm to calculate the optimal meeting time. For speech recognition technology, Azure Speech Service and Google Speech-to-Text API are used. This technology transcribes speech during meetings into text in real time and automatically generates meeting minutes based on the content of the speech.

[0480] The server automatically generates an agenda based on the meeting's objectives using a generation AI model. This utilizes a template-based generation AI model, allowing the AI ​​to suggest appropriate agenda items based on prompts such as "Please create an agenda for the market launch of a new product."

[0481] Usage example

[0482] Users can access the platform from their devices and view meeting details. For example, the server's suggested dates and times for the next meeting are generated based on user feedback. With a prompt such as, "Please tell me the best meeting time considering everyone's availability," the server uses AI to suggest the most suitable options.

[0483] This system streamlines the entire process, from meeting preparation and execution to scheduling the next meeting, allowing users to focus more on the discussion. Data generated at each stage of the process is continuously analyzed by the server, contributing to improved quality in subsequent meetings. In this way, it is possible to achieve both increased efficiency and improved quality throughout the entire meeting process.

[0484] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0485] Step 1:

[0486] The server retrieves schedule information from the API of the scheduling management system used by each participant. This input data contains information about each participant's schedule. The server collects this data and stores it in a database to list the free time of all participants. As part of the data processing, it runs an algorithm to remove overlapping time and non-work time.

[0487] Step 2:

[0488] The server compares and analyzes the availability of all participants based on the stored schedule data. The input data is the schedule information processed in Step 1. The server calculates and proposes the optimal meeting time. In this process, a generative AI model is used, and prompts such as "Please tell me the optimal meeting time considering the availability of all participants" are used to have the AI ​​calculate the recommended time. The optimal meeting time is generated as output.

[0489] Step 3:

[0490] The server receives the meeting objective and automatically generates a corresponding agenda using an AI model. The input at this stage is information about the meeting objective. It selects and constructs appropriate topics and subjects using past meeting records and templates. The output is the automatically generated agenda distributed to participants. A specific example of its operation is using the prompt, "Please create agenda items regarding the market launch of the new product."

[0491] Step 4:

[0492] During the meeting, the server captures audio in real time and uses speech recognition technology to transcribe the speech into text. The input data is the audio data collected during the meeting. The audio data is analyzed, separated by speaker, and output as text data. The system operates by using a speech recognition API (e.g., Azure Speech Service) to perform simultaneous text transcription.

[0493] Step 5:

[0494] The server automatically generates meeting minutes based on text data obtained through speech recognition. The input data is the text data generated in step 4. The server summarizes this data, extracts key points, and structures it as meeting minutes. The output is the meeting minutes, which are shared after the meeting. The server has the function to distribute the meeting minutes electronically to participants.

[0495] Step 6:

[0496] The server organizes the action items decided during the meeting and creates a task list for each person in charge. This uses the meeting minutes created in step 5 as input. The action items are notified along with the person in charge and the deadline, and the output is an individual task list. The terminal displays this task list to the user, allowing them to track and check the progress.

[0497] Step 7:

[0498] The server collects feedback from participants to optimize the date and time of the next meeting. The input data is feedback collected from participants. Based on this, a generative AI model and the prompt "Please tell me the best date and time for the next meeting" are used to calculate and suggest candidate dates and times. The output is the adjusted candidate date and time for the next meeting.

[0499] Step 8:

[0500] The server meticulously analyzes the speech data collected during and after the meeting. The input data consists of speech data from all meeting participants. The server analyzes the frequency of speech and the flow of discussion to identify areas for improvement. The output is provided to participants as a report to help improve the quality of future meetings. A data analysis algorithm is used in its operation.

[0501] (Application Example 1)

[0502] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0503] While efficiency improvements are needed in data center and other facility operations, such as meetings and maintenance, coordinating participant schedules, creating agendas, and managing meeting minutes still require considerable time and effort. Furthermore, facility maintenance and procurement of necessary parts are predominantly managed manually, often leading to inefficiencies. In this context, there is a need to achieve efficiency and automation across the entire process of meeting management and facility management.

[0504] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0505] In this invention, the server includes means for acquiring participants' schedule information and proposing the optimal meeting time; means for automatically generating meeting items based on the purpose and communicating them to participants; means for analyzing audio during the meeting and transcribing speech in real time; means for automatically generating a record based on the content of speech and sharing it with participants after the meeting; means for linking the status of equipment with maintenance activity schedules and proposing the optimal timing for maintenance activities; and means for automatically ordering necessary equipment and parts. This reduces the burden of meeting management and enables efficient equipment management.

[0506] "Participants" are people who are involved in a meeting or activity and who have the role of providing schedules or opinions.

[0507] "Meeting time" refers to the time when participants gather to exchange information and make decisions.

[0508] An "agenda" is a list of topics and content to be addressed during a meeting, and it indicates the purpose and direction of the conference.

[0509] "Speech analysis" refers to the technology that acquires spoken words as audio data and converts it into text information.

[0510] "Transcribing speech" is the process of recording spoken content as digital text.

[0511] "Records" refer to documents or data that preserve the content of discussions during a meeting in a way that allows for later reference.

[0512] "Maintenance activities" refer to periodic maintenance and repair activities carried out to maintain the normal operation of equipment and systems.

[0513] "Equipment and parts" refers to the physical elements and replaceable parts necessary for the operation of the equipment.

[0514] "Automated ordering" is a process that orders necessary goods based on predetermined conditions, with minimal human involvement.

[0515] This system is built around a multi-functional server. The server utilizes cloud services, such as AWS Lambda and DynamoDB, to efficiently process data. The server can retrieve participants' schedule information from the Google Calendar API, analyze it, and automatically suggest the optimal meeting time. This information is then distributed to participants via smartphones and computers.

[0516] Furthermore, the server uses generative AI models such as OpenAI's GPT-3 to generate an agenda based on the purpose of the meeting. This makes it possible to notify participants of the meeting's flow in advance and encourage them to prepare. During the meeting, Google Cloud Speech-to-Text is used to transcribe speeches in real time, and this data is stored in DynamoDB.

[0517] Furthermore, after the meeting, the server summarizes key points based on accumulated data and individually notifies participants of specific action items. The server also calculates the optimal timing for maintenance activities based on the equipment status and maintenance schedule, and automatically places orders for necessary equipment and parts in advance.

[0518] For example, in a maintenance meeting for cooling systems within a data center, the server can suggest the optimal repair time and place orders considering parts inventory. A possible prompt request might be, "Please suggest the optimal date for the next cooling system repair, taking into account the team members' schedules."

[0519] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0520] Step 1:

[0521] The server retrieves participants' schedule information via the Google Calendar API. The input is each participant's calendar access permission, and the server collects their schedule data based on this. The output is schedule data including the free time slots of all participants. Based on this, the server calculates the intersection of free time slots and proposes the optimal meeting time.

[0522] Step 2:

[0523] The server uses the collected data and leverages OpenAI's GPT-3 generative AI model to automatically generate an agenda based on the meeting's objectives. Inputs include the meeting's theme and past meeting minutes. Output is a detailed meeting agenda sent to participants. The server distributes this agenda to devices via email or a dedicated application.

[0524] Step 3:

[0525] During the meeting, the server uses Google Cloud Speech-to-Text to transcribe participants' speech in real time. The input is audio data collected during the meeting, and the output is transcribed text data. The server stores this data in DynamoDB for later processing.

[0526] Step 4:

[0527] After the meeting concludes, the server analyzes the accumulated text data and summarizes the key points. The input is the transcribed and saved meeting minutes, and the output is the summarized meeting minutes. The server uses this summary to send each participant a to-do list of action items.

[0528] Step 5:

[0529] The server analyzes equipment status and maintenance logs to plan the optimal timing for maintenance activities. Inputs include equipment operating status data and repair history data. Outputs are specific activity schedules notified to the maintenance team. Furthermore, the server predicts necessary equipment and parts and places purchase orders through an automated ordering system.

[0530] Step 6:

[0531] The user uses prompts to send details about the next meeting and other specific requests to the generating AI model. The input is a text-based request from the user (e.g., "Please suggest the best date for the next cooling system repair, taking into account the team members' schedules"). The output is the optimal suggestion generated by the AI ​​and displayed on the user's device.

[0532] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0533] This invention is a system designed to improve the quality of meetings by not only streamlining meeting management but also analyzing the emotional state of participants. This system utilizes artificial intelligence, combining speech recognition and emotion analysis to grasp participants' emotions in real time and provide feedback to facilitate the meeting.

[0534] The central server of the system receives audio data from meeting participants and first uses a speech recognition engine to transcribe their speech into text in real time. Simultaneously, an emotion engine analyzes speech patterns and linguistic expressions to detect emotional states. For example, it can identify positive, negative, or neutral emotions based on the tone and speed of speech, as well as specific keywords.

[0535] By performing sentiment analysis in real time, the server sends instructions to the terminal to appropriately adjust the progress of the meeting. The terminal can then notify the user, for example, "Participant A may be losing interest" or "The agenda item seems to have suddenly elicited a negative reaction," prompting adjustments to the ongoing agenda.

[0536] After the meeting, the server analyzes the data obtained by the emotion engine and generates a detailed report summarizing emotional trends and responses to specific topics, which is then distributed to participants. This provides material for concretely considering improvements for the next meeting and supports better decision-making.

[0537] For example, if participants' emotions regarding a particular product topic show a generally negative trend during a meeting, the emotional data will be used to consider areas for product improvement or switching to a different topic. This feedback will be provided after the meeting and used to prepare for the next meeting.

[0538] This system enables the server to manage meetings dynamically based on emotional data, and makes it easier for users to grasp the overall atmosphere of the meeting. This increases the likelihood of improved satisfaction for all participants and increased meeting productivity.

[0539] The following describes the processing flow.

[0540] Step 1:

[0541] The server accesses each participant's calendar API to retrieve their schedule information and collects their availability. Based on this, it identifies the optimal meeting time and proposes it to all participants.

[0542] Step 2:

[0543] The server automatically generates an agenda based on the meeting's objectives. The generated agenda is then distributed to participants via email or a dedicated application.

[0544] Step 3:

[0545] Once the meeting begins, the terminal sends the audio data from the meeting to the server. The server uses a speech recognition engine to transcribe the spoken words into text in real time.

[0546] Step 4:

[0547] The server has an emotion engine built in that analyzes audio data to identify the emotional state of participants. It assigns positive, negative, and neutral emotion tags.

[0548] Step 5:

[0549] Based on the results of sentiment analysis, the server evaluates the progress of the meeting in real time and suggests adjusting the agenda to the user via the terminal at the appropriate time.

[0550] Step 6:

[0551] After the meeting ends, the server generates meeting minutes and comprehensively analyzes participants' reactions based on sentiment data. This then generates a detailed sentiment report.

[0552] Step 7:

[0553] The server distributes the analysis results and sentiment reports to all participants, helping them understand areas for improvement for the next meeting.

[0554] Step 8:

[0555] Users review the reports sent from the server and use the meeting's outcomes and improvement suggestions to prepare for the next meeting.

[0556] (Example 2)

[0557] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0558] In today's business environment, there is a demand for improved meeting efficiency and productivity. In particular, insufficient understanding of participants' emotions regarding discussions can lead to a decline in the quality of decision-making and a decrease in the effectiveness of meetings. Furthermore, the lack of means to analyze emotions and provide feedback in real time makes it difficult to manage meetings flexibly according to the situation.

[0559] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0560] In this invention, the server includes means for receiving audio signals and analyzing language data and emotional states; means for evaluating participants' responses based on the analyzed emotional states and adjusting the progress of the meeting; and means for aggregating emotional analysis data and generating a report for evaluating the quality of the meeting. This makes it possible to accurately grasp the emotional shifts of participants during and after the meeting, and to provide feedback and adjust the progress of the meeting based on that.

[0561] "Participant schedule information" refers to data about the time available to meeting attendees, including their free time and existing appointments.

[0562] The "method for suggesting meeting times" is a function that calculates and presents the optimal meeting start time based on the participants' schedule information.

[0563] A "meeting schedule" is a plan that outlines the agenda and the order in which the meeting will proceed, based on its objectives.

[0564] An "audio signal" is data obtained by electrically converting sound received through a microphone or similar device.

[0565] "Language data" refers to information about characters and phrases extracted from speech signals using speech recognition technology.

[0566] "Emotional state" refers to information about an individual's emotional response, obtained through an analysis of their tone of voice and word choice.

[0567] "Means of adjusting the progress" refers to a function that dynamically adjusts the pace of discussion and topics based on information gathered during a meeting.

[0568] "Emotional analysis data" refers to numerical data representing statistics and trends related to emotions, analyzed based on participants' statements.

[0569] The "means of generating reports" refer to a function that uses data accumulated after a meeting to create a report summarizing the content of the meeting and the results of an analysis of the emotions expressed.

[0570] This invention provides a system that enhances the quality of meeting management and precisely analyzes the emotional state of participants. The system consists of a server and terminals, with the central server integrating speech recognition technology and emotion analysis technology. The following describes how this system is implemented.

[0571] The server uses commercially available speech recognition software as its speech recognition engine. Examples of such software include Google Cloud Speech-to-Text and Amazon Transcribe. The server utilizes this software to receive audio signals from meeting participants in real time and convert them into language data.

[0572] Furthermore, Microsoft Azure Text Analytics and IBM Watson Tone Analyzer can be used as sentiment analysis engines. These allow for the inference of participants' emotional states from linguistic data. The server collects the acquired sentiment data, evaluates participants' emotional trends in real time, and adjusts the program accordingly.

[0573] For example, the server uses sentiment analysis to determine that "Participant A is showing a negative reaction to the presentation" and sends a notification to the device. The device then provides feedback to the user, such as prompting them to move on to the next agenda item.

[0574] In addition, after the meeting ends, the server integrates all the collected data and generates a report summarizing the sentiment analysis results for the entire meeting. For example, by entering the prompt "Analyze the sentiments of meeting participants regarding the latest product presentation and summarize their impressions," an analysis based on the generated AI model will be performed.

[0575] This allows users to understand participants' emotions and consider reviewing and improving the content of future meetings. Through this system, improvements in meeting productivity and participant satisfaction are expected.

[0576] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0577] Step 1:

[0578] The server receives audio signals from meeting participants. These audio signals are acquired through microphones and other means and are first converted into digital data. The input is an analog audio signal, and the output is digital audio data. This makes it possible to process the audio using a speech recognition engine.

[0579] Step 2:

[0580] The server sends digital audio data to a speech recognition process, which converts it to text. Speech recognition software is used here, for example, Google Cloud Speech-to-Text. The input is digital audio data, and the output is real-time text data. Specifically, a person's speech is transcribed sequentially by the computer.

[0581] Step 3:

[0582] The server sends the converted text data to the sentiment analysis engine. Using tools such as Microsoft Azure Text Analytics, the engine analyzes the text data to determine the emotional state. The input is text data, and the output is data indicating the emotional state. At this stage, specific keywords and sentence tones are analyzed to identify the participant's emotions at that time.

[0583] Step 4:

[0584] Based on the analysis results, the server sends meeting progress adjustment suggestions to the terminal. It evaluates the sentiment analysis data and generates feedback that takes into account the participants' level of interest and emotional trends. The input is emotional state data, and the output is information related to progress adjustments. Specifically, a notification such as "Negative reactions to agenda item A are increasing" is conveyed to the user via the terminal.

[0585] Step 5:

[0586] After the meeting concludes, the server summarizes all the analysis data and compiles the meeting's sentiment trends into a final report. The report is automatically generated based on prompts generated by a generative AI model. The input is the accumulated sentiment analysis data, and the output is a meeting evaluation report. This report provides users with concrete information to help them prepare for and strategize for future meetings.

[0587] (Application Example 2)

[0588] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] Traditional meeting systems were inefficient because meeting progress management and agenda generation were done manually. Furthermore, the lack of real-time monitoring of participants' comments and emotional states meant that meeting quality was easily influenced by participants' subjective opinions. In addition, in service industries, understanding customers' emotional states and improving responses accordingly was difficult. This hindered improvements in meeting productivity and customer satisfaction.

[0590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0591] In this invention, the server includes means for acquiring time management information of participants and proposing the optimal meeting time, means for analyzing emotional states during conversations and notifying participants of their level of interest and emotional responses in real time, and means used during conversations with customers to support improvements in customer service based on the customer's emotional state. This enables not only increased efficiency and improved quality of meetings, but also the optimization of customer service in physical stores.

[0592] "Time management information" refers to data that includes participants' schedules and appointments, and is used to optimize the timing of meetings and other activities.

[0593] An "agenda" refers to a plan that lists the topics or themes to be discussed in a meeting or discussion.

[0594] "Documenting speech" is the process of converting spoken words into text for recording, and it is important for later review and analysis of what was said in a meeting.

[0595] A "task list" is a list of specific tasks or activities that need to be completed, and it is notified individually to the participants or those responsible.

[0596] "Emotional state" refers to individual psychological reactions and feelings that can be interpreted from voice and facial expressions, and is classified through emotional analysis.

[0597] "Customer service improvement" refers to initiatives and processes aimed at improving the quality of service and interaction with customers.

[0598] The system for implementing this invention processes data in real time by combining speech recognition and sentiment analysis technologies. The server receives audio data from the microphone of a smartphone or smart glasses. This audio data is converted to text using a speech recognition engine in the cloud (e.g., Google Cloud Speech-to-Text API). Next, sentiment analysis is performed on this text data using a natural language processing engine (e.g., IBM Watson Natural Language Understanding).

[0599] Furthermore, the server sends real-time notifications to users based on the analysis results. These notifications provide feedback tailored to the participants' level of interest and emotions, helping to improve meeting flow and customer service. Specifically, if a customer shows interest, the user can make suggestions that emphasize that point.

[0600] A concrete example is when a system detects that a customer is showing interest in a new product and has a positive emotional response to it while it's being introduced in a physical store. Based on this information, the system can then provide the user with more detailed information about the product or encourage them to purchase it.

[0601] An example of a prompt for a generative AI model is: "During a conversation with a customer, analyze the customer's emotions from the following audio data. Determine whether they are positive, negative, or neutral and notify us in real time."

[0602] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0603] Step 1:

[0604] The device uses a microphone to collect ambient sound in real time. The audio data is recorded as a digital signal and sent to the server as input data.

[0605] Step 2:

[0606] The server sends the received audio data to the speech recognition engine. This engine uses the Google Cloud Speech-to-Text API to convert the audio into text data. Data processing is performed to make the audio signal a format that can be processed by language. The converted text becomes the input for the next step.

[0607] Step 3:

[0608] The server sends the text data to the sentiment analysis engine. This engine uses IBM Watson Natural Language Understanding to extract emotional states from the text. Specifically, it analyzes keywords and tone within the text to determine emotional categories such as positive, negative, and neutral. The resulting emotional data is the output for the next step.

[0609] Step 4:

[0610] The server generates appropriate feedback based on the analysis results. It sends prompts to the generating AI model to create feedback messages that match the situation. These prompts are based on the analyzed sentiment data and may include content such as, "The customer appears interested in the product."

[0611] Step 5:

[0612] The device notifies the user of feedback messages. These notifications enable the user to take appropriate actions and improvements in real time, depending on the situation. Prompt messages are displayed visually or audibly through information dissemination tools to inform the user.

[0613] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0614] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0615] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0616] [Fourth Embodiment]

[0617] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0618] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0619] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0620] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0621] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0622] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0623] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0624] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0625] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0626] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0627] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0628] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0629] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0630] The meeting support system according to the present invention is designed to embody a series of processes for efficiently operating and managing meetings throughout the pre-meeting preparation, during the meeting, and after the meeting. Its embodiments are described in detail below.

[0631] The core of this system lies in a server, which collects data from the APIs of the scheduling management systems used by each participant to obtain their schedule information. The server analyzes the collected data and automatically suggests the optimal meeting time by comparing the availability of all participants.

[0632] The server then automatically generates an agenda based on the meeting's objectives. It utilizes past meeting records and purpose-specific templates to set appropriate topics. This generated agenda is then communicated to participants via email or a dedicated application, allowing them to review it before the meeting.

[0633] During the meeting, the server uses speech recognition technology to transcribe participants' statements in real time. The server analyzes the audio data, identifies speakers, and automatically generates meeting minutes. This feature allows participants to focus on the discussion and reduces the effort required for record-keeping.

[0634] After the meeting, the server automatically generates meeting minutes, summarizes them, extracts key points, and distributes them to participants. It also organizes the action items decided during the meeting and notifies the responsible parties with specific deadlines. During this process, the terminal displays a to-do list to the user, allowing them to manage their progress.

[0635] Furthermore, when scheduling the next meeting, the server collects feedback from participants and, based on that, proposes the most suitable dates and times again. Iterating through this process promotes further efficiency.

[0636] The data from meetings is analyzed in detail by a server, recording things like the frequency of each speaker and the flow of topics during the meeting. Based on this data, points for improvement in future meetings are suggested. These suggestions are distributed to each participant as a dedicated report, serving as foundational material for improving the quality of meetings.

[0637] For example, if the server sets the meeting theme to "Market launch of a new product," it will automatically generate an agenda aligned with that theme and send it to the participants. As the meeting progresses, the server will record each statement and guide the participants to efficiently carry out the next steps.

[0638] In this way, servers, terminals, and users work together within the entire system, enabling efficient meeting management. In particular, the server's data analysis and optimization algorithms contribute to simplifying meeting preparation and reducing the burden on participants.

[0639] The following describes the processing flow.

[0640] Step 1:

[0641] The server retrieves schedule information from participants' schedule management systems via API. All participants' schedules are stored in a database and prepared for analysis.

[0642] Step 2:

[0643] The server compares each participant's schedule and extracts common available time slots. This allows it to list the optimal meeting times that all participants can attend.

[0644] Step 3:

[0645] The server automatically generates an agenda based on the meeting's objectives, using past meeting records and preset templates. The generated agenda is then sent to all participants via email or a dedicated app.

[0646] Step 4:

[0647] When a meeting begins, the terminal sends the meeting audio to the server. The server uses a speech recognition engine to transcribe the audio into text in real time, creating the basic data for the meeting minutes.

[0648] Step 5:

[0649] The server analyzes the transcribed data, organizes the content for each speaker, and incorporates it into the meeting minutes. Simultaneously, it monitors the progress of the meeting and manages time according to the agenda. It issues instructions to move on to the next topic as needed.

[0650] Step 6:

[0651] After the meeting ends, the server automatically summarizes the generated minutes and extracts key decisions and to-do lists. Based on this, it sends a report to participants via email.

[0652] Step 7:

[0653] The server organizes the action items decided during the meeting by person in charge, sets specific deadlines, and notifies each participant via their terminal as a to-do list.

[0654] Step 8:

[0655] The server adjusts the next meeting date based on feedback collected from participants. It presents participants with proposed dates and times to facilitate the scheduling process.

[0656] Step 9:

[0657] The server analyzes the data collected during the meeting to evaluate the frequency of participation and the progress of the discussion. Based on the analysis results, improvement suggestions are compiled into a report and provided to the participants.

[0658] (Example 1)

[0659] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0660] In today's business environment, efficient meeting management is a critical challenge. Especially when many participants are involved, significant effort is required to optimize meeting times, create agendas, and record and share meeting content. Furthermore, it's essential to properly manage the action items decided upon after the meeting and use them to improve future meetings. Traditional systems fail to efficiently address these challenges, making it difficult to improve the quality of meetings.

[0661] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0662] In this invention, the server includes means for acquiring participants' schedule information and suggesting the optimal time, means for automatically generating an agenda based on the purpose of the meeting and distributing it to participants, and means for analyzing speech and transcribing speech into text in real time. As a result, meeting preparation time is significantly reduced, participants can concentrate on the discussion, post-meeting action item management is streamlined, and the overall quality of the meeting can be improved.

[0663] "Participant schedule information" refers to data on the schedules and available times of individuals participating in the meeting.

[0664] "Suggesting the optimal time" is the process of selecting the most appropriate and efficient meeting start time, taking into account the schedules of all participants.

[0665] A "meeting agenda based on the purpose of the meeting" is a list of topics or subjects set in line with the focus and goals to be achieved at the meeting.

[0666] "Automatic generation" refers to a system creating content based on predetermined algorithms and data, without manual human intervention.

[0667] "Analyzing speech and converting it to text in real time" refers to the process of instantly converting spoken words during a meeting into text information using speech recognition technology.

[0668] "Automatically generating records" means that the system independently organizes the collected data and creates records based on that data.

[0669] A "list of individual tasks" is a list that summarizes the tasks and responsibilities decided during the meeting for each participant.

[0670] "Automatically adjusting the next meeting date" is a process that re-examines the availability of meeting participants and suggests the optimal date and time for the next meeting.

[0671] "Analyzing data and making improvement suggestions based on participants' comments" means analyzing recorded comment data, identifying areas for improvement in meeting management methods, and proposing solutions.

[0672] "Generating a document describing technical features" involves creating a document that summarizes the meeting content from a technical perspective, based on meeting records and key points.

[0673] This conference support system is designed to facilitate efficient communication between servers, terminals, and users. Specific implementations using hardware and software are described below.

[0674] hardware

[0675] The server retrieves participants' schedule information using the APIs of each scheduling management system. Specifically, it collects each participant's available time using the Google Calendar API and the Microsoft Graph API. The server uses a high-performance server computing unit suitable for processing large amounts of data.

[0676] software

[0677] The system software running on the server analyzes the acquired schedule data and executes an algorithm to calculate the optimal meeting time. For speech recognition technology, Azure Speech Service and Google Speech-to-Text API are used. This technology transcribes speech during meetings into text in real time and automatically generates meeting minutes based on the content of the speech.

[0678] The server automatically generates an agenda based on the meeting's objectives using a generation AI model. This utilizes a template-based generation AI model, allowing the AI ​​to suggest appropriate agenda items based on prompts such as "Please create an agenda for the market launch of a new product."

[0679] Usage example

[0680] Users can access the platform from their devices and view meeting details. For example, the server's suggested dates and times for the next meeting are generated based on user feedback. With a prompt such as, "Please tell me the best meeting time considering everyone's availability," the server uses AI to suggest the most suitable options.

[0681] This system streamlines the entire process, from meeting preparation and execution to scheduling the next meeting, allowing users to focus more on the discussion. Data generated at each stage of the process is continuously analyzed by the server, contributing to improved quality in subsequent meetings. In this way, it is possible to achieve both increased efficiency and improved quality throughout the entire meeting process.

[0682] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0683] Step 1:

[0684] The server retrieves schedule information from the API of the scheduling management system used by each participant. This input data contains information about each participant's schedule. The server collects this data and stores it in a database to list the free time of all participants. As part of the data processing, it runs an algorithm to remove overlapping time and non-work time.

[0685] Step 2:

[0686] The server compares and analyzes the availability of all participants based on the stored schedule data. The input data is the schedule information processed in Step 1. The server calculates and proposes the optimal meeting time. In this process, a generative AI model is used, and prompts such as "Please tell me the optimal meeting time considering the availability of all participants" are used to have the AI ​​calculate the recommended time. The optimal meeting time is generated as output.

[0687] Step 3:

[0688] The server receives the meeting objective and automatically generates a corresponding agenda using an AI model. The input at this stage is information about the meeting objective. It selects and constructs appropriate topics and subjects using past meeting records and templates. The output is the automatically generated agenda distributed to participants. A specific example of its operation is using the prompt, "Please create agenda items regarding the market launch of the new product."

[0689] Step 4:

[0690] During the meeting, the server captures audio in real time and uses speech recognition technology to transcribe the speech into text. The input data is the audio data collected during the meeting. The audio data is analyzed, separated by speaker, and output as text data. The system operates by using a speech recognition API (e.g., Azure Speech Service) to perform simultaneous text transcription.

[0691] Step 5:

[0692] The server automatically generates meeting minutes based on text data obtained through speech recognition. The input data is the text data generated in step 4. The server summarizes this data, extracts key points, and structures it as meeting minutes. The output is the meeting minutes, which are shared after the meeting. The server has the function to distribute the meeting minutes electronically to participants.

[0693] Step 6:

[0694] The server organizes the action items decided during the meeting and creates a task list for each person in charge. This uses the meeting minutes created in step 5 as input. The action items are notified along with the person in charge and the deadline, and the output is an individual task list. The terminal displays this task list to the user, allowing them to track and check the progress.

[0695] Step 7:

[0696] The server collects feedback from participants to optimize the date and time of the next meeting. The input data is feedback collected from participants. Based on this, a generative AI model and the prompt "Please tell me the best date and time for the next meeting" are used to calculate and suggest candidate dates and times. The output is the adjusted candidate date and time for the next meeting.

[0697] Step 8:

[0698] The server meticulously analyzes the speech data collected during and after the meeting. The input data consists of speech data from all meeting participants. The server analyzes the frequency of speech and the flow of discussion to identify areas for improvement. The output is provided to participants as a report to help improve the quality of future meetings. A data analysis algorithm is used in its operation.

[0699] (Application Example 1)

[0700] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0701] While efficiency improvements are needed in data center and other facility operations, such as meetings and maintenance, coordinating participant schedules, creating agendas, and managing meeting minutes still require considerable time and effort. Furthermore, facility maintenance and procurement of necessary parts are predominantly managed manually, often leading to inefficiencies. In this context, there is a need to achieve efficiency and automation across the entire process of meeting management and facility management.

[0702] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0703] In this invention, the server includes means for acquiring participants' schedule information and proposing the optimal meeting time; means for automatically generating meeting items based on the purpose and communicating them to participants; means for analyzing audio during the meeting and transcribing speech in real time; means for automatically generating a record based on the content of speech and sharing it with participants after the meeting; means for linking the status of equipment with maintenance activity schedules and proposing the optimal timing for maintenance activities; and means for automatically ordering necessary equipment and parts. This reduces the burden of meeting management and enables efficient equipment management.

[0704] "Participants" are people who are involved in a meeting or activity and who have the role of providing schedules or opinions.

[0705] "Meeting time" refers to the time when participants gather to exchange information and make decisions.

[0706] An "agenda" is a list of topics and content to be addressed during a meeting, and it indicates the purpose and direction of the conference.

[0707] "Speech analysis" refers to the technology that acquires spoken words as audio data and converts it into text information.

[0708] "Transcribing speech" is the process of recording spoken content as digital text.

[0709] "Records" refer to documents or data that preserve the content of discussions during a meeting in a way that allows for later reference.

[0710] "Maintenance activities" refer to periodic maintenance and repair activities carried out to maintain the normal operation of equipment and systems.

[0711] "Equipment and parts" refers to the physical elements and replaceable parts necessary for the operation of the equipment.

[0712] "Automated ordering" is a process that orders necessary goods based on predetermined conditions, with minimal human involvement.

[0713] This system is built around a multi-functional server. The server utilizes cloud services, such as AWS Lambda and DynamoDB, to efficiently process data. The server can retrieve participants' schedule information from the Google Calendar API, analyze it, and automatically suggest the optimal meeting time. This information is then distributed to participants via smartphones and computers.

[0714] Furthermore, the server uses generative AI models such as OpenAI's GPT-3 to generate an agenda based on the purpose of the meeting. This makes it possible to notify participants of the meeting's flow in advance and encourage them to prepare. During the meeting, Google Cloud Speech-to-Text is used to transcribe speeches in real time, and this data is stored in DynamoDB.

[0715] Furthermore, after the meeting, the server summarizes key points based on accumulated data and individually notifies participants of specific action items. The server also calculates the optimal timing for maintenance activities based on the equipment status and maintenance schedule, and automatically places orders for necessary equipment and parts in advance.

[0716] For example, in a maintenance meeting for cooling systems within a data center, the server can suggest the optimal repair time and place orders considering parts inventory. A possible prompt request might be, "Please suggest the optimal date for the next cooling system repair, taking into account the team members' schedules."

[0717] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0718] Step 1:

[0719] The server retrieves participants' schedule information via the Google Calendar API. The input is each participant's calendar access permission, and the server collects their schedule data based on this. The output is schedule data including the free time slots of all participants. Based on this, the server calculates the intersection of free time slots and proposes the optimal meeting time.

[0720] Step 2:

[0721] The server uses the collected data and leverages OpenAI's GPT-3 generative AI model to automatically generate an agenda based on the meeting's objectives. Inputs include the meeting's theme and past meeting minutes. Output is a detailed meeting agenda sent to participants. The server distributes this agenda to devices via email or a dedicated application.

[0722] Step 3:

[0723] During the meeting, the server uses Google Cloud Speech-to-Text to transcribe participants' speech in real time. The input is audio data collected during the meeting, and the output is transcribed text data. The server stores this data in DynamoDB for later processing.

[0724] Step 4:

[0725] After the meeting concludes, the server analyzes the accumulated text data and summarizes the key points. The input is the transcribed and saved meeting minutes, and the output is the summarized meeting minutes. The server uses this summary to send each participant a to-do list of action items.

[0726] Step 5:

[0727] The server analyzes equipment status and maintenance logs to plan the optimal timing for maintenance activities. Inputs include equipment operating status data and repair history data. Outputs are specific activity schedules notified to the maintenance team. Furthermore, the server predicts necessary equipment and parts and places purchase orders through an automated ordering system.

[0728] Step 6:

[0729] The user uses prompts to send details about the next meeting and other specific requests to the generating AI model. The input is a text-based request from the user (e.g., "Please suggest the best date for the next cooling system repair, taking into account the team members' schedules"). The output is the optimal suggestion generated by the AI ​​and displayed on the user's device.

[0730] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0731] This invention is a system designed to improve the quality of meetings by not only streamlining meeting management but also analyzing the emotional state of participants. This system utilizes artificial intelligence, combining speech recognition and emotion analysis to grasp participants' emotions in real time and provide feedback to facilitate the meeting.

[0732] The central server of the system receives audio data from meeting participants and first uses a speech recognition engine to transcribe their speech into text in real time. Simultaneously, an emotion engine analyzes speech patterns and linguistic expressions to detect emotional states. For example, it can identify positive, negative, or neutral emotions based on the tone and speed of speech, as well as specific keywords.

[0733] By performing sentiment analysis in real time, the server sends instructions to the terminal to appropriately adjust the progress of the meeting. The terminal can then notify the user, for example, "Participant A may be losing interest" or "The agenda item seems to have suddenly elicited a negative reaction," prompting adjustments to the ongoing agenda.

[0734] After the meeting, the server analyzes the data obtained by the emotion engine and generates a detailed report summarizing emotional trends and responses to specific topics, which is then distributed to participants. This provides material for concretely considering improvements for the next meeting and supports better decision-making.

[0735] For example, if participants' emotions regarding a particular product topic show a generally negative trend during a meeting, the emotional data will be used to consider areas for product improvement or switching to a different topic. This feedback will be provided after the meeting and used to prepare for the next meeting.

[0736] This system enables the server to manage meetings dynamically based on emotional data, and makes it easier for users to grasp the overall atmosphere of the meeting. This increases the likelihood of improved satisfaction for all participants and increased meeting productivity.

[0737] The following describes the processing flow.

[0738] Step 1:

[0739] The server accesses each participant's calendar API to retrieve their schedule information and collects their availability. Based on this, it identifies the optimal meeting time and proposes it to all participants.

[0740] Step 2:

[0741] The server automatically generates an agenda based on the meeting's objectives. The generated agenda is then distributed to participants via email or a dedicated application.

[0742] Step 3:

[0743] Once the meeting begins, the terminal sends the audio data from the meeting to the server. The server uses a speech recognition engine to transcribe the spoken words into text in real time.

[0744] Step 4:

[0745] The server has an emotion engine built in that analyzes audio data to identify the emotional state of participants. It assigns positive, negative, and neutral emotion tags.

[0746] Step 5:

[0747] Based on the results of sentiment analysis, the server evaluates the progress of the meeting in real time and suggests adjusting the agenda to the user via the terminal at the appropriate time.

[0748] Step 6:

[0749] After the meeting ends, the server generates meeting minutes and comprehensively analyzes participants' reactions based on sentiment data. This then generates a detailed sentiment report.

[0750] Step 7:

[0751] The server distributes the analysis results and sentiment reports to all participants, helping them understand areas for improvement for the next meeting.

[0752] Step 8:

[0753] Users review the reports sent from the server and use the meeting's outcomes and improvement suggestions to prepare for the next meeting.

[0754] (Example 2)

[0755] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0756] In today's business environment, there is a demand for improved meeting efficiency and productivity. In particular, insufficient understanding of participants' emotions regarding discussions can lead to a decline in the quality of decision-making and a decrease in the effectiveness of meetings. Furthermore, the lack of means to analyze emotions and provide feedback in real time makes it difficult to manage meetings flexibly according to the situation.

[0757] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0758] In this invention, the server includes means for receiving audio signals and analyzing language data and emotional states; means for evaluating participants' responses based on the analyzed emotional states and adjusting the progress of the meeting; and means for aggregating emotional analysis data and generating a report for evaluating the quality of the meeting. This makes it possible to accurately grasp the emotional shifts of participants during and after the meeting, and to provide feedback and adjust the progress of the meeting based on that.

[0759] "Participant schedule information" refers to data about the time available to meeting attendees, including their free time and existing appointments.

[0760] The "method for suggesting meeting times" is a function that calculates and presents the optimal meeting start time based on the participants' schedule information.

[0761] A "meeting schedule" is a plan that outlines the agenda and the order in which the meeting will proceed, based on its objectives.

[0762] An "audio signal" is data obtained by electrically converting sound received through a microphone or similar device.

[0763] "Language data" refers to information about characters and phrases extracted from speech signals using speech recognition technology.

[0764] "Emotional state" refers to information about an individual's emotional response, obtained through an analysis of their tone of voice and word choice.

[0765] "Means of adjusting the progress" refers to a function that dynamically adjusts the pace of discussion and topics based on information gathered during a meeting.

[0766] "Emotional analysis data" refers to numerical data representing statistics and trends related to emotions, analyzed based on participants' statements.

[0767] The "means of generating reports" refer to a function that uses data accumulated after a meeting to create a report summarizing the content of the meeting and the results of an analysis of the emotions expressed.

[0768] This invention provides a system that enhances the quality of meeting management and precisely analyzes the emotional state of participants. The system consists of a server and terminals, with the central server integrating speech recognition technology and emotion analysis technology. The following describes how this system is implemented.

[0769] The server uses commercially available speech recognition software as its speech recognition engine. Examples of such software include Google Cloud Speech-to-Text and Amazon Transcribe. The server utilizes this software to receive audio signals from meeting participants in real time and convert them into language data.

[0770] Furthermore, Microsoft Azure Text Analytics and IBM Watson Tone Analyzer can be used as sentiment analysis engines. These allow for the inference of participants' emotional states from linguistic data. The server collects the acquired sentiment data, evaluates participants' emotional trends in real time, and adjusts the program accordingly.

[0771] For example, the server uses sentiment analysis to determine that "Participant A is showing a negative reaction to the presentation" and sends a notification to the device. The device then provides feedback to the user, such as prompting them to move on to the next agenda item.

[0772] In addition, after the meeting ends, the server integrates all the collected data and generates a report summarizing the sentiment analysis results for the entire meeting. For example, by entering the prompt "Analyze the sentiments of meeting participants regarding the latest product presentation and summarize their impressions," an analysis based on the generated AI model will be performed.

[0773] This allows users to understand participants' emotions and consider reviewing and improving the content of future meetings. Through this system, improvements in meeting productivity and participant satisfaction are expected.

[0774] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0775] Step 1:

[0776] The server receives audio signals from meeting participants. These audio signals are acquired through microphones and other means and are first converted into digital data. The input is an analog audio signal, and the output is digital audio data. This makes it possible to process the audio using a speech recognition engine.

[0777] Step 2:

[0778] The server sends digital audio data to a speech recognition process, which converts it to text. Speech recognition software is used here, for example, Google Cloud Speech-to-Text. The input is digital audio data, and the output is real-time text data. Specifically, a person's speech is transcribed sequentially by the computer.

[0779] Step 3:

[0780] The server sends the converted text data to the sentiment analysis engine. Using tools such as Microsoft Azure Text Analytics, the engine analyzes the text data to determine the emotional state. The input is text data, and the output is data indicating the emotional state. At this stage, specific keywords and sentence tones are analyzed to identify the participant's emotions at that time.

[0781] Step 4:

[0782] Based on the analysis results, the server sends meeting progress adjustment suggestions to the terminal. It evaluates the sentiment analysis data and generates feedback that takes into account the participants' level of interest and emotional trends. The input is emotional state data, and the output is information related to progress adjustments. Specifically, a notification such as "Negative reactions to agenda item A are increasing" is conveyed to the user via the terminal.

[0783] Step 5:

[0784] After the meeting concludes, the server summarizes all the analysis data and compiles the meeting's sentiment trends into a final report. The report is automatically generated based on prompts generated by a generative AI model. The input is the accumulated sentiment analysis data, and the output is a meeting evaluation report. This report provides users with concrete information to help them prepare for and strategize for future meetings.

[0785] (Application Example 2)

[0786] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0787] Traditional meeting systems were inefficient because meeting progress management and agenda generation were done manually. Furthermore, the lack of real-time monitoring of participants' comments and emotional states meant that meeting quality was easily influenced by participants' subjective opinions. In addition, in service industries, understanding customers' emotional states and improving responses accordingly was difficult. This hindered improvements in meeting productivity and customer satisfaction.

[0788] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0789] In this invention, the server includes means for acquiring time management information of participants and proposing the optimal meeting time, means for analyzing emotional states during conversations and notifying participants of their level of interest and emotional responses in real time, and means used during conversations with customers to support improvements in customer service based on the customer's emotional state. This enables not only increased efficiency and improved quality of meetings, but also the optimization of customer service in physical stores.

[0790] "Time management information" refers to data that includes participants' schedules and appointments, and is used to optimize the timing of meetings and other activities.

[0791] An "agenda" refers to a plan that lists the topics or themes to be discussed in a meeting or discussion.

[0792] "Documenting speech" is the process of converting spoken words into text for recording, and it is important for later review and analysis of what was said in a meeting.

[0793] A "task list" is a list of specific tasks or activities that need to be completed, and it is notified individually to the participants or those responsible.

[0794] "Emotional state" refers to individual psychological reactions and feelings that can be interpreted from voice and facial expressions, and is classified through emotional analysis.

[0795] "Customer service improvement" refers to initiatives and processes aimed at improving the quality of service and interaction with customers.

[0796] The system for implementing this invention processes data in real time by combining speech recognition and sentiment analysis technologies. The server receives audio data from the microphone of a smartphone or smart glasses. This audio data is converted to text using a speech recognition engine in the cloud (e.g., Google Cloud Speech-to-Text API). Next, sentiment analysis is performed on this text data using a natural language processing engine (e.g., IBM Watson Natural Language Understanding).

[0797] Furthermore, the server sends real-time notifications to users based on the analysis results. These notifications provide feedback tailored to the participants' level of interest and emotions, helping to improve meeting flow and customer service. Specifically, if a customer shows interest, the user can make suggestions that emphasize that point.

[0798] A concrete example is when a system detects that a customer is showing interest in a new product and has a positive emotional response to it while it's being introduced in a physical store. Based on this information, the system can then provide the user with more detailed information about the product or encourage them to purchase it.

[0799] An example of a prompt for a generative AI model is: "During a conversation with a customer, analyze the customer's emotions from the following audio data. Determine whether they are positive, negative, or neutral and notify us in real time."

[0800] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0801] Step 1:

[0802] The device uses a microphone to collect ambient sound in real time. The audio data is recorded as a digital signal and sent to the server as input data.

[0803] Step 2:

[0804] The server sends the received audio data to the speech recognition engine. This engine uses the Google Cloud Speech-to-Text API to convert the audio into text data. Data processing is performed to make the audio signal a format that can be processed by language. The converted text becomes the input for the next step.

[0805] Step 3:

[0806] The server sends the text data to the sentiment analysis engine. This engine uses IBM Watson Natural Language Understanding to extract emotional states from the text. Specifically, it analyzes keywords and tone within the text to determine emotional categories such as positive, negative, and neutral. The resulting emotional data is the output for the next step.

[0807] Step 4:

[0808] The server generates appropriate feedback based on the analysis results. It sends prompts to the generating AI model to create feedback messages that match the situation. These prompts are based on the analyzed sentiment data and may include content such as, "The customer appears interested in the product."

[0809] Step 5:

[0810] The device notifies the user of feedback messages. These notifications enable the user to take appropriate actions and improvements in real time, depending on the situation. Prompt messages are displayed visually or audibly through information dissemination tools to inform the user.

[0811] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0812] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0813] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0814] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0815] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0816] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0817] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0818] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0819] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0820] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0821] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0822] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0823] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0824] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0825] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0826] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0827] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0828] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0829] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0830] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0831] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0832] The following is further disclosed regarding the embodiments described above.

[0833] (Claim 1)

[0834] A method for obtaining participants' schedule information and suggesting the optimal meeting time,

[0835] A means of automatically generating and distributing a meeting agenda based on its purpose to participants,

[0836] A method for analyzing audio during meetings and transcribing speech into text in real time,

[0837] A method for automatically generating meeting minutes based on the content of the discussions and sharing them with participants after the meeting,

[0838] A method for organizing the action items decided at the meeting and notifying the person in charge of an individual to-do list,

[0839] A method to automatically adjust the date and time of the next meeting and propose it to participants,

[0840] A method for analyzing meeting data and making improvement suggestions based on participants' comments,

[0841] A system that includes this.

[0842] (Claim 2)

[0843] The system according to claim 1, which appropriately manages the progress of a discussion based on speaking time and prompts a transition to the next agenda item as needed.

[0844] (Claim 3)

[0845] The system according to claim 1, which optimizes candidate dates and times for the next meeting based on feedback from participants.

[0846] "Example 1"

[0847] (Claim 1)

[0848] A device that obtains participants' schedule information and suggests the optimal time,

[0849] A device that automatically generates an agenda based on the purpose of the meeting and distributes it to participants,

[0850] A device that analyzes speech and transcribes it into text in real time,

[0851] A device that automatically generates a record based on the content of the speech and later shares it with the participants,

[0852] A device that organizes the decided action items and notifies the person in charge of a list of individual tasks,

[0853] A device that automatically adjusts the next period and proposes it to participants,

[0854] A device that analyzes data and makes improvement suggestions based on the participants' comments,

[0855] A device that generates a text describing technical features using meeting information,

[0856] A system that includes this.

[0857] (Claim 2)

[0858] The system according to claim 1, which appropriately manages the progress based on speaking time and facilitates the transition to the next agenda item.

[0859] (Claim 3)

[0860] The system according to claim 1, which optimizes candidate dates and times for the next meeting based on feedback from participants.

[0861] "Application Example 1"

[0862] (Claim 1)

[0863] A method for obtaining participants' schedule information and suggesting the optimal meeting time,

[0864] A means of automatically generating meeting items based on the purpose and communicating them to participants,

[0865] A method for analyzing audio during a meeting and transcribing speech in real time,

[0866] A means of automatically generating a record based on the content of the speeches and sharing it with participants after the meeting,

[0867] A means of organizing the action items decided at the meeting and notifying the person in charge of an individual task list,

[0868] A method to automatically adjust the date and time of the next meeting and propose it to participants,

[0869] A method for analyzing meeting data and making improvement suggestions based on participants' comments,

[0870] A means of linking the status of equipment with maintenance activity plans to propose the optimal timing for maintenance activities,

[0871] A means of automatically ordering necessary equipment and parts,

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, which appropriately manages the progress of a discussion based on speaking time and prompts a transition to the next agenda item as needed.

[0875] (Claim 3)

[0876] The system according to claim 1, which optimizes candidate dates and times for the next meeting based on feedback from participants.

[0877] "Example 2 of combining an emotion engine"

[0878] (Claim 1)

[0879] A method for obtaining participants' schedule information and suggesting the optimal meeting time,

[0880] A method for automatically generating and distributing a meeting agenda based on the objectives to participants,

[0881] A method for analyzing audio during meetings and transcribing statements in real time,

[0882] A method for automatically generating meeting minutes based on the content of the discussions and sharing them with participants after the meeting,

[0883] A means of organizing the action items decided at the meeting and notifying the person in charge of an individual task list,

[0884] A method to automatically adjust the date and time of the next meeting and propose it to participants,

[0885] A means for receiving audio signals and analyzing language data and emotional states,

[0886] A means of evaluating participants' responses based on their analyzed emotional states and adjusting the progress accordingly,

[0887] A means for aggregating sentiment analysis data and generating a report to evaluate the quality of a meeting,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, which appropriately manages the progress of a discussion based on speaking time and prompts a transition to the next agenda item as needed.

[0891] (Claim 3)

[0892] The system according to claim 1, which optimizes candidate dates and times for the next meeting based on feedback from participants.

[0893] "Application example 2 when combining with an emotional engine"

[0894] (Claim 1)

[0895] A method for obtaining participants' time management information and suggesting the optimal meeting time,

[0896] A method for automatically creating and distributing a meeting schedule based on objectives to participants,

[0897] A method for analyzing speech during a conversation and documenting the utterance in real time,

[0898] A means of automatically generating a record document based on the content of the discussion and sharing it with participants after the meeting,

[0899] A means of organizing the action items decided at the meeting and notifying the person in charge of an individual task list,

[0900] A method to automatically adjust the time of the next meeting and propose it to participants,

[0901] A method for analyzing meeting data and making improvement suggestions based on participants' comments,

[0902] A means of analyzing emotional states during a conversation and notifying participants of their level of interest and emotional responses in real time,

[0903] A means used during conversations with customers to support the improvement of customer service based on the customer's emotional state,

[0904] A system that includes this.

[0905] (Claim 2)

[0906] The system according to claim 1, which appropriately manages the progress of a discussion based on speaking time and prompts a transition to the next agenda item as needed.

[0907] (Claim 3)

[0908] The system according to claim 1, which optimizes candidate times for the next meeting based on feedback from participants. [Explanation of Symbols]

[0909] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A method for obtaining participants' schedule information and suggesting the optimal meeting time, A means of automatically generating meeting items based on the purpose and communicating them to participants, A method for analyzing audio during a meeting and transcribing speech in real time, A means of automatically generating a record based on the content of the speeches and sharing it with participants after the meeting, A means of organizing the action items decided at the meeting and notifying the person in charge of an individual task list, A method to automatically adjust the date and time of the next meeting and propose it to participants, A method for analyzing meeting data and making improvement suggestions based on participant comments, A means of linking the status of equipment with maintenance activity plans to propose the optimal timing for maintenance activities, A means of automatically ordering necessary equipment and parts, A system that includes this.

2. The system according to claim 1, which appropriately manages the progress of a discussion based on the speaking time and prompts a transition to the next agenda item as needed.

3. The system according to claim 1, which optimizes candidate dates and times for the next meeting based on feedback from participants.

Images