system

Abstract

Provide a system. A method for acquiring collected conference log data; means for extracting important points from the meeting log data and generating a summary; means for analyzing said summaries to identify progress and emerging issues; means for updating or adding events based on said progress and new challenges; means for notifying a user of the updated or added event information; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In project management, sharing information and checking progress in meetings is important, but it is often difficult to properly summarize meeting content or accurately grasp progress and new issues. As a result, project management becomes inefficient and early problem detection and resolution are delayed. To solve these issues, a system is needed that can quickly and accurately identify progress and new issues through automatic collection and detailed analysis of meeting log data, and manage them as appropriate events. [Means for solving the problem]

[0005] The present invention provides a system that automatically collects meeting log data, extracts key points, and generates summaries. Specifically, the system solves problems in project management by including a means for acquiring collected meeting log data, a means for extracting key points from the meeting log data and generating summaries, a means for analyzing the summaries to identify progress and new issues, a means for updating or adding events based on the progress and new issues, and a means for notifying users of the updated or added event information. This system uses natural language processing technology to analyze the meeting log data and quickly identifies progress and new issues and provides them to users, thereby improving the efficiency of project management.

[0006] "Meeting log data" refers to all data that records the contents of a meeting, including the audio, text, participant list, start time, and end time of the meeting.

[0007] "Extraction" refers to the act of selecting important information or points from meeting log data.

[0008] A "summary" refers to a text that concisely summarizes the contents of the meeting based on the extracted important information.

[0009] "Progress" refers to the current state of progress of a project or task.

[0010] "New issues" refer to new problems or issues that arise during the meeting and need to be resolved.

[0011] An "event" is something within a project management system that represents the progress or response of a particular task or issue.

[0012] "User" refers to an individual or group who uses the system to manage projects.

[0013] "Collection means" refers to a method or device for capturing conference log data into the system.

[0014] "Natural language processing technology" refers to technology that enables computers to understand and analyze human language.

[0015] "Notification means" refers to a method or device by which the system notifies the user of information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention provides a system for supporting efficient management of conference contents, and a specific implementation method of the system based on the roles of the server, terminal, and user will be described below.

[0038] 1. Collecting meeting log data

[0039] When a user holds a meeting, the server automatically collects the meeting log data. For example, it uses the API of the online meeting platform to obtain the content of comments made during the meeting and participant information. This allows the server to collect meeting log data that includes information such as the meeting audio, text, participant list, start time, and end time.

[0040] 2. Summary and analysis of the meeting content

[0041] The server takes the collected meeting log data and converts it into text using natural language processing technology, then extracts key points from the text data and generates a concise summary.

[0042] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing well, but Team B is facing a lack of resources."

[0043] The server then uses this summary data to track progress and identify emerging issues, making it clear whether the project is progressing as planned or where problems lie.

[0044] 3. Event updates and additions

[0045] The terminal automatically updates events on the project management system based on the progress and task information received from the server, reflecting the progress status in existing events and adding new task-based events as necessary.

[0046] For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "Lack of resources."

[0047] 4. User Notification and Response

[0048] The device notifies the user of updated events and new challenges, allowing the user to quickly understand progress and take action to address any issues that arise.

[0049] For example, a user can receive a notification, log in to a project management system to view the details, and then consider how to address the resource shortage issue and take appropriate action.

[0050] Specific examples

[0051] Suppose a user holds an online project progress meeting. After the meeting ends, the server collects the meeting log data (audio, text, participant list, etc.) through the API of the online meeting platform.

[0052] The server analyzes the collected log data, generates a summary such as "Project A is progressing smoothly, but Team B is short on resources," and based on this summary, identifies the issue of "shortage of resources for Team B."

[0053] The terminal updates the "Project A Progress" event in the project management system and adds a new event called "Lack of resources."

[0054] The terminal notifies the user of these updates, and the user can log in and check the details. The user can take measures to address resource shortages and continue the smooth progress of the project.

[0055] In this way, the system based on the present invention realizes efficient project management through the summarization and analysis of meeting contents.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] Users use the online meeting platform to start a project status meeting, during which all statements and participant information are recorded.

[0059] Step 2:

[0060] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, including audio data, text chat, participant list, start time, end time, etc.

[0061] Step 3:

[0062] The server stores the acquired meeting log data in a database, which makes it available for later analysis.

[0063] Step 4:

[0064] The server uses natural language processing technology to convert the voice data into text, which includes everything said during the meeting.

[0065] Step 5:

[0066] The server analyzes the text data and performs processing to extract key points, for example, identifying important keywords and phrases to use as the basis for summarizing.

[0067] Step 6:

[0068] The server generates a summary of the meeting based on the extracted key points, including progress reports, issues, and new topics.

[0069] Step 7:

[0070] The server then analyzes the generated summary to identify progress and new issues, for example, "Project A is progressing smoothly, but Team B is facing a resource shortage."

[0071] Step 8:

[0072] The terminal updates events on the project management system based on the progress status and new task information received from the server, adding progress information to existing events and generating events for new tasks.

[0073] Step 9:

[0074] The device notifies the user of updated event information, including a summary of progress and new tasks, keeping the user informed quickly and accurately.

[0075] Step 10:

[0076] Users receive notifications and log into the project management system to view details, and initiate appropriate action based on the progress and issue information provided.

[0077] These are the specific processing steps of the program. This system allows users to efficiently manage the progress of projects and quickly respond to new challenges.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] Conventional meeting management systems have made it difficult to effectively grasp important comments and issues raised during meetings and reflect them in subsequent project management. They also lacked a way to quickly notify users of meeting progress and new issues. This can lead to important information being overlooked during project progress, affecting the quality and progress of the final deliverables.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server

[0083] A means of obtaining the collected meeting log data using the API of the online meeting platform;

[0084] means for converting the conference log data from voice data to text data using natural language processing technology;

[0085] means for extracting key points from the text data to generate summaries and identify progress and emerging issues;

[0086] A means for updating or adding events to a project management system based on the progress and new issues;

[0087] means for notifying a user of the updated or added event information;

[0088] This makes it possible to efficiently collect and analyze important meeting information and reflect it in the project management system. Furthermore, by promptly notifying users of progress and new issues, the project can be progressed appropriately and problems can be resolved.

[0089] "Conference log data" refers to information recorded during an online conference, such as speech content, voice data, text chat, participant list, and conference time.

[0090] An "online conference platform" is software or a service that allows multiple users to hold conferences over the Internet while sharing audio and video in real time.

[0091] "API" stands for Application Program Interface, an interface for sharing information and functions between different software systems.

[0092] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language, and is used to analyze text and voice data.

[0093] "Text data" is voice data converted into character information, and is used to facilitate analysis and display.

[0094] A "summary" is a short summary of important points extracted from the original text data.

[0095] "Progress" is information that indicates how far a project or the like has progressed relative to a plan.

[0096] An "issue" is a problem or obstacle that needs to be resolved as the project progresses.

[0097] A "project management system" is a system that centrally manages project planning, progress management, task allocation, etc., and supports efficient project management.

[0098] An "event" is an item that indicates information such as a specific progress or issue in the project management system.

[0099] "Notification" is a means by which a system notifies a user of specific information, and includes alerts and messages.

[0100] This invention is a system that supports efficient management of online meeting contents by effectively collecting and analyzing log data of online meetings and reflecting it in the related project management system. This system mainly operates based on the roles of the server, terminal, and user.

[0101] When a user starts a conference using the API of an online conference platform (e.g., an online conference application), the server automatically collects conference log data. This log data includes speech content (audio data), text chat, a participant list, the start and end times of the conference, etc. The audio data is processed to identify the speaker using speaker diarization technology.

[0102] After collecting the log data, the server stores it in storage and converts the voice data into text data using natural language processing technology. Specifically, it uses Google® Cloud Speech-to-Text API or IBM Watson® Speech to Text. The converted text data is then analyzed using Google Cloud Natural Language API or a generative AI model (e.g., OpenAI®'s GPT-4®) to extract key points. The resulting summary might include, for example, "Project A is progressing smoothly, but Team B is facing a resource shortage." Based on this summary, the server identifies progress and new challenges.

[0103] The identified progress and issue information is sent from the server to the device. The device receives this information and automatically updates the event using the API of the project management system (e.g., project management software). Specifically, the new progress is reflected in the "Project A Progress" event, and an event based on the new issue, "Lack of Resources," is added. This process is performed through the device's API request, and the updated event information is saved in the project management system.

[0104] Updated and added event information is notified to the user via the terminal. The user can then log in to the project management system to check detailed information and take specific measures, such as allocating additional staff to resolve resource shortages.

[0105] Specific examples

[0106] Consider a case where a user holds an online project progress meeting. After the meeting ends, the server automatically collects the meeting log data (audio, text, participant list, meeting time, etc.) using the API of the online meeting platform. At this time, speaker diarization is applied to identify the speaker.

[0107] The server stores the collected log data in storage and converts the voice data into text using the Google Cloud Speech-to-Text API. The text data is then analyzed using OpenAI's GPT-4, which generates a summary that says, "Project A is progressing smoothly, but Team B's lack of resources is a problem," and identifies the "lack of resources" as a new issue.

[0108] The device receives the data sent from the server and uses the API of the project management software to update the "Project A Progress" event and add a new event called "Lack of Resources." For example, it sends an API request to add "Team B's resource shortage is an issue" to the event description.

[0109] The device notifies the user of these updates, who can then log into the project management software to view the details and take concrete action, such as allocating additional staff to address resource shortages.

[0110] Prompt Sentence Examples

[0111] Here is an example prompt for a generative AI model to explain the system in more detail:

[0112] "Please describe in detail a system that supports efficient management of meeting content. The system collects log data from online meetings and uses natural language processing techniques to summarize and identify issues. Based on this, the system updates events in a project management system and notifies users. Please also mention the specific APIs and analysis tools used."

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Step 1:

[0115] When a user starts an online conference, the server uses the API of the online conference platform to obtain conference log data. The obtained data includes speech content (audio data), text chat, participant list, and conference start and end times. Specifically, this data is obtained through API calls and saved in internal storage. The input is the online conference session ID, and the output is temporary saved conference log data.

[0116] Step 2:

[0117] The server converts the acquired voice data into text data using natural language processing technology. It then uses the Google Cloud Speech-to-Text API to convert the voice data into text. The input is voice data, and the output is text data. Specifically, it sends voice data to the API and receives the returned text data.

[0118] Step 3:

[0119] The server analyzes the converted text data, extracts key points, and generates a summary using the Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4). The input is the text data, and the output is a summary, identified progress, and new challenges. Specifically, the server sends the text data to the analysis engine and receives the analysis results.

[0120] Step 4:

[0121] The server organizes the identified progress and new issues along with the summary, and sends the organized data to the device. The input is the analyzed summary data, and the output is notification data to the device. Specifically, the server sends the organized data to the device's API.

[0122] Step 5:

[0123] The device automatically updates the event using the API of the project management system based on the received data. It sends an API request to update the "Project A Progress" event with the latest progress information, and generates a new event based on the new issue of "Lack of resources." The input is progress information and issue information from the server, and the output is an updated event in the project management system. Specifically, it sends an API request to update the event details.

[0124] Step 6:

[0125] The terminal notifies the user of the updated event information. The user receives the notification and logs in to the project management system to check the detailed information. The input is the event information of the project management system, and the output is the notification to the user. The specific operation is to generate a notification message and send it to the user's device.

[0126] Step 7:

[0127] Upon receiving the notification, the user accesses the project management system and takes specific measures. For example, they may take action such as adding new staff to resolve resource shortages. The input is the notified event information, and the output is the implementation of specific measures. Specifically, the user logs in to the project management system, checks the details, and records the measures.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] Currently, managing meeting content at manufacturing sites requires a lot of manual work, and recording and summarizing the content after meetings often takes a lot of time. This delays understanding progress and identifying new issues, making it difficult to respond quickly. Furthermore, with on-site work being hectic, there are limitations to accurately recording meeting content and quickly notifying relevant parties of the necessary information. To solve this situation, there is a need for a system that automatically collects and analyzes meeting logs and quickly notifies relevant parties of summaries and issues.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes: means for acquiring collected meeting log data; means for extracting important points from the meeting log data and generating a summary; means for analyzing the summary and identifying progress and new issues; means for updating or adding events based on the progress and new issues; means for notifying the user of the updated or added event information; means installed in the smart glasses for recording the meeting log in real time; and means for automatically analyzing the meeting log to generate a summary and notifying the user of the progress and new issues. This improves the efficiency of meeting content management at factory manufacturing sites, enabling quick progress understanding and issue identification.

[0133] "Collected conference log data" refers to data that includes information such as audio recorded during a conference, text, a participant list, start time, and end time.

[0134] "Important points" refer to statements, issues, decisions, etc. that are considered particularly important within the content of the meeting.

[0135] A "summary" is a document or textual information that briefly summarizes what was discussed at a meeting.

[0136] "Progress" refers to the current state or stage of a project or task.

[0137] "New issues" refer to new problems or issues that need to be addressed that were identified during the meeting.

[0138] "Updating or adding an event" means modifying or creating new tasks or schedules in an existing project management system based on the latest information.

[0139] "Notifying users" means informing users of the generated update information and information about new issues via email, a notification system, or the like.

[0140] "Smart glasses" are a type of wearable device that refers to a glasses-type display device that has the ability to display and record information in real time.

[0141] "Real-time recording" means collecting audio and video simultaneously while the meeting is in progress and saving the data.

[0142] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[0143] The present invention relates to a system for efficiently managing meetings at a manufacturing site in a factory. The system is configured based on the roles of smart glasses, a server, a terminal, and a user.

[0144] Meeting log data collection

[0145] First, when a user holds a meeting, the smart glasses record the meeting log data in real time. The smart glasses are equipped with speech recognition and text conversion functions, and automatically collect speech data during the meeting and convert it into text data. This data is then sent to a server.

[0146] Meeting summary and analysis

[0147] The server receives the meeting log data sent from the smart glasses and converts it into text using natural language processing technology. It then extracts key points from the collected text data and generates a concise summary. The server uses the SpaCy model as its natural language processing technology. For example, it generates a summary such as, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0148] Event updates and additions

[0149] The terminal automatically updates events on the project management system based on the progress and issue information received from the server. It reflects the progress status in existing events and adds events based on new issues as necessary. For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "lack of resources."

[0150] User notification and response

[0151] The device notifies the user of updated events and new tasks, allowing the user to quickly understand the progress and take action to address any issues that arise. Specifically, email notifications are sent using Python's smtplib. For example, notifications are sent to email addresses such as "user1@example.com" and "user2@example.com."

[0152] Specific examples

[0153] For example, a regular meeting is held in the manufacturing department of a factory, and after the meeting, smart glasses collect the meeting log and generate a summary such as "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials." Based on this, a new issue is added to the project management system and the user is notified about it by email.

[0154] Prompt Sentence Examples

[0155] Generate a meeting log stating, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0156] This approach streamlines meeting content management on the factory floor, allowing for quick progress updates and the identification of emerging issues, while also enabling more advanced analysis and automation using generative AI models and prompts.

[0157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0158] Step 1:

[0159] Collection and transmission of meeting log data

[0160] A user wears smart glasses and holds a conference. The glasses collect audio data during the conference in real time and convert it into text data. This text data includes the content of the speech, the speaker, a timestamp, etc. The smart glasses then send this data to a server.

[0161] Input: Audio data during the meeting

[0162] Output: Meeting log data in text format

[0163] Step 2:

[0164] Receiving and storing meeting log data

[0165] The server receives the meeting log data sent from the smart glasses and stores it in a database for later analysis.

[0166] Input: Text data sent from smart glasses

[0167] Output: Meeting log data stored in a database

[0168] Step 3:

[0169] Meeting summary generation

[0170] The server analyzes the saved meeting log data using natural language processing technology. It uses a generative AI model (e.g., SpaCy) to extract key points and generate a summary. For example, a summary might be generated that reads, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0171] Input: Meeting log data stored in a database

[0172] Output: Summary of meeting content

[0173] Step 4:

[0174] Identifying progress and new challenges

[0175] The server identifies progress and new issues from the generated summary, and uses natural language processing techniques to extract problem and progress information from keywords and phrases in the summary.

[0176] Input: Summary of meeting content

[0177] Output: Identified progress and challenges

[0178] Step 5:

[0179] Event updates and additions

[0180] The terminal automatically updates events in the project management system based on the identified progress and issue information. It reflects the progress in existing events and adds new events based on issues as needed. For example, it adds a new resource arrangement event based on the fact that "production line 2 is short of raw materials."

[0181] Input: Identified progress and challenges

[0182] Output: Updated or added event information

[0183] Step 6:

[0184] User Notification

[0185] The device notifies the user of updated events and new challenges via email or push notifications on the smartphone app, allowing the user to quickly understand progress and address any issues.

[0186] Input: Updated or added event information

[0187] Output: User notification

[0188] These steps allow for efficient management of meeting content and enable rapid response on-site.

[0189] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0190] This invention combines a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support efficient management of meeting content with an emotion engine that recognizes user emotions. Below, we will explain specific implementation methods of this invention based on the roles of the server, terminal, and user.

[0191] 1. Collecting meeting log data

[0192] Users use the online meeting platform to hold project progress meetings, and all statements, text chats, and participant information are recorded by the platform.

[0193] 2. Acquiring meeting log data

[0194] After the meeting ends, the server retrieves the meeting log data through the online meeting platform's API. The retrieved data includes audio data, text data, participant list, start time, and end time. The retrieved meeting log data is then stored in a database.

[0195] 3. Summary and analysis of the meeting content

[0196] The server uses natural language processing technology to convert the voice data into text, analyzes the converted text data, extracts important points, and generates a summary of the meeting content based on the extracted points.

[0197] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing smoothly, but Team B is experiencing a resource shortage."

[0198] 4. Emotion Recognition by Emotion Engine

[0199] During the analysis of the meeting log data, the server uses an emotion engine to recognize emotions from the user's remarks and attitudes. For example, it can determine whether the user is feeling stressed from the pitch of their voice and the choice of words.

[0200] The perceived emotional information is reflected in the meeting summary, and additional information is added, for example, in the form of "Team B's leader expresses strong concerns about the lack of resources."

[0201] 5. Event updates and additions

[0202] The terminal automatically updates and adds events to the project management system based on the progress, new issues, and emotion information sent from the server. For example, the terminal adds the latest progress report to the "Project A Progress" event and generates an event for the new issue of "Lack of resources."

[0203] 6. User Notification and Response

[0204] The device notifies the user of updated event information, including progress, new tasks, and user emotional information, at the most appropriate time.

[0205] For example, it is considered that by notifying the user when he or she is relaxed, the user can calmly deal with the problem.

[0206] Specific examples

[0207] Suppose a user holds an online project progress meeting. After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform.

[0208] The server analyzes the log data and converts it into text using natural language processing technology. It extracts key points and generates a summary such as, "Project A is progressing smoothly, but Team B is facing a lack of resources."

[0209] Furthermore, the server uses an emotion engine to extract emotional information from the log data and add information such as "Team B members are feeling very stressed about the lack of resources."

[0210] The device updates the events on the project management system based on the received information, generates a new event for the issue "resource shortage," and sends a notification to the user while taking into account emotional information.

[0211] Users receive notifications, log in to their project management systems to review the details, consider how to address the resource shortage, and initiate appropriate actions.

[0212] The present invention enables efficient management of meeting content including emotional information, and enables prompt and accurate responses to problems.

[0213] The processing flow will be explained below.

[0214] Step 1:

[0215] Users use an online meeting platform to initiate a project status meeting, during which all statements, text chats, and participant information are recorded.

[0216] Step 2:

[0217] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, which includes audio data, text data, a list of participants, and the start and end times.

[0218] Step 3:

[0219] The server stores the acquired log data in a database so that it can be used for later analysis.

[0220] Step 4:

[0221] The server uses natural language processing technology to convert the voice data into text, which includes all the content spoken during the meeting.

[0222] Step 5:

[0223] The server analyzes the converted text data, extracts important points and keywords from the meeting, and generates a summary based on the extracted points.

[0224] Step 6:

[0225] The server uses the summary to identify progress and emerging issues, such as "Project A is progressing well, but Team B is facing a lack of resources."

[0226] Step 7:

[0227] The server uses an emotion engine to recognize users' emotions from meeting log data, for example, determining whether they are stressed based on their voice pitch and word choice.

[0228] Step 8:

[0229] The server adds the recognized emotional information to the meeting summary, which includes emotional information such as "Team B members are highly stressed about the lack of resources."

[0230] Step 9:

[0231] The terminal automatically updates and adds events to the project management system based on the progress status, new tasks, and emotion information received from the server. Existing events are updated with the latest progress information, and new events are generated for new tasks.

[0232] Step 10:

[0233] The device notifies the user of updated event information, including progress, new tasks, and emotional information, and the content and timing of notifications are tailored to the user's emotional state.

[0234] Step 11:

[0235] Users receive notifications, log into their project management system to review details, and initiate appropriate action based on the progress, issue information, and sentiment information provided.

[0236] These are the specific processing steps based on the present invention, which combines an emotion engine. This system allows users to efficiently manage project progress and quickly respond to new challenges. In addition, taking emotion information into account promotes optimal behavior for problem solving.

[0237] Example 2

[0238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] Conventional meeting management systems have difficulty efficiently grasping the progress and issues of meetings, and more accurate decision-making is required, especially by including participants' emotional information. Furthermore, since summarizing meeting content and extracting key points takes time and effort, automation is necessary. Furthermore, project management updates and notifications based on emotional information are also an important issue.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0241] In this invention, the server includes means for acquiring collected meeting log data, means for converting the meeting log data into voice data and text data, means for extracting important points from the voice data using natural language processing technology and generating a summary, means for extracting user emotional information from the text data and voice data using emotion analysis technology and incorporating the information into the summary, means for analyzing the summary to identify progress and new issues, means for updating or adding events to the project management system based on the progress and new issues, and means for notifying the user of the updated or added event information and emotional information. This enables efficient management of meeting content, accelerated decision-making including emotional information, and automatic updating and notification of the project management system.

[0242] "Conference log data" refers to data recorded during a conference, including statements made during the conference, text chat, participant information, and the start and end times of the conference.

[0243] "Audio data" refers to data recorded in audio file format of what was said during a conference.

[0244] "Text data" refers to text-format data converted from voice data, or text chat data recorded during a conference.

[0245] "Natural language processing technology" is a technology for analyzing language data and understanding its meaning, and in the present invention, it is a technology used as a means for converting voice data into text and extracting important points.

[0246] "Emotion analysis technology" is a technology that recognizes and extracts the emotional state of a speaker from text data and audio data.

[0247] The "summary" is a concise summary of the important points of the meeting content extracted using natural language processing technology.

[0248] "Progress" refers to the current state or progress of a project or task.

[0249] "Issues" are problems that arise during the project or areas that require improvement.

[0250] An "event" is an entry in a project management system that records and tracks actions on progress or new issues.

[0251] A "project management system" is a software system for planning, executing, and monitoring tasks or projects.

[0252] "Communication means" refers to the technology and methods by which servers and terminals exchange data with online meeting platforms and project management systems.

[0253] "Notification" is the act of notifying the user of updated event information or important messages.

[0254] The present invention provides a system for efficiently managing meeting log data collected using an online meeting platform and generating a detailed summary including user emotional information. In the following embodiments, specific implementation methods will be described based on the roles of the server, terminal, and user.

[0255] 1. Overview of the entire system

[0256] This system supports large-scale project management and efficient meeting management for remote teams. The system acquires meeting log data from an online meeting platform and analyzes it using natural language processing and sentiment analysis technologies. As a result, it is possible to grasp a summary of the meeting content, progress, new issues, and user sentiment. This automatically updates events in the project management system and sends notifications to users at optimal times.

[0257] 2. Hardware and Software Configuration

[0258] server

[0259] The server is a general-purpose computer with a powerful processor and sufficient memory, running the following software:

[0260] Operating system: Linux (registered trademark)

[0261] Database: MySQL (registered trademark)

[0262] Natural language processing technology: Google Cloud Speech-to-Text API, spaCy

[0263] Emotion analysis technology: Amazon Comprehend

[0264] Terminal

[0265] The terminal is a personal computer or mobile device operated by a user, running the following software:

[0266] Operating Systems: Windows, macOS (registered trademark), ANDROID (registered trademark), iOS

[0267] Project management systems: JIRA, Asana

[0268] User

[0269] Users are project managers and meeting participants who participate in meetings through an online meeting platform. In particular, the following environment is required:

[0270] Online meeting platform: Zoom, MICROSOFT(R) TEAMS(R)

[0271] 3. Specific examples of implementation

[0272] A user holds a project status meeting using Zoom. After the meeting ends, the server retrieves the meeting log data through the Zoom API. The server converts the acquired audio data into text using the Google Cloud Speech-to-Text API, analyzes the converted text data using natural language processing technology (spaCy), and extracts key points. The server then extracts sentiment information using Amazon Comprehend and combines the summary with the sentiment information.

[0273] Specific prompt examples:

[0274] "Generate a summary of a project status meeting. Analyze the following log data and include key points and sentiment information."

[0275] 4. Operational Flow

[0276] A user holds a meeting, and the meeting log data is recorded by the online meeting platform. After the meeting ends, the server retrieves the recorded data and performs natural language processing and sentiment analysis. A summary is generated based on the analysis results, and the event is updated and added to the project management system. The terminal notifies the user of the update, and the user can access the project management system to check the details.

[0277] The present invention enables efficient management of meeting content and quick and accurate responses that take into account user feelings, thereby increasing the success rate of projects.

[0278] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0279] Step 1: Hosting a meeting and collecting log data

[0280] A user uses an online conference platform to hold a project progress meeting. When the user starts the meeting, the online conference platform records the remarks made during the meeting, text chat, and participant information in real time. The input is the user's voice and text remarks, and the output is the meeting log data. This log data is stored on the server of the online conference platform.

[0281] Step 2: Retrieving meeting log data

[0282] After a meeting ends, the server retrieves the meeting log data through the API of the online meeting platform. Specifically, the server sends a request to the API endpoint to retrieve the meeting log data, including audio data, text data, a list of participants, and the start and end times. The input is the request to the API, and the output is the retrieved dataset. This dataset is then stored in a database.

[0283] Step 3: Transcribe the meeting content

[0284] The server converts the audio data into text using the Google Cloud Speech-to-Text API. The server sends the acquired audio data to the API and receives the corresponding text data. The input is audio data and the output is text data. This text data is saved as a temporary file and used in the next step.

[0285] Step 4: Analyze text data and generate summaries

[0286] The server uses an NLP (Natural Language Processing) library (e.g. spaCy) to analyze the converted text data and extract key points. The server analyzes the text data and extracts the main points of the meeting. The input is the text data, and the output is a list of key points. A summary is generated based on these key points. The server aggregates the extracted points and generates a summary of the meeting content. The output is the generated summary.

[0287] Step 5: Performing Emotion Recognition

[0288] The server sends text and audio data to the emotion engine Amazon Comprehend to extract the user's emotional information. The server passes the acquired data to the emotion engine and receives the analysis results. The input is text and audio data, and the output is emotional information. This emotional information is added to the summary. For example, information such as "The leader of Team B expresses strong concerns about the lack of resources" is added.

[0289] Step 6: Update your project management system

[0290] The device updates the project management system based on the progress, new tasks, and sentiment information sent from the server. The device receives data from the server and updates and adds events to the project management system through API. The input is the data received from the server, and the output is the updated event in the project management system. For example, a new report is added to "Project A Progress."

[0291] Step 7: Notify users

[0292] The device notifies the user of the updated event information. Based on the updated information, the device sends a notification to the user in an appropriate format (e.g., push notification, email, dashboard display). The input is the updated event information, and the output is a notification to the user. The user receives the notification and logs in to the project management system to check the detailed information. For example, specific instructions such as "Please start considering countermeasures for the resource shortage issue" are displayed.

[0293] The system enables efficient management of meeting content, generation of detailed summaries including emotional information, and automatic updates and notifications for project management systems.

[0294] (Application example 2)

[0295] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0296] In production management and maintenance meetings at factories, it is necessary to efficiently grasp progress and identify new issues. However, typical meeting recording systems are limited to summarizing meetings and extracting progress, and are unable to consider the emotions and stress levels of meeting participants. This poses a challenge, making it difficult to provide more detailed and practical problem-solving based on emotional information. In particular, in factories, where resource shortages and emergency responses are often required, a rapid response that takes into account the emotions of participants is required.

[0297] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0298] In this invention, the server includes means for acquiring collected meeting log data, means for extracting important points from the meeting log data and generating a summary, means for recognizing emotions from user comments and attitudes, means for reflecting the emotional information in the summary, means for updating or adding events based on the progress and new tasks, and means for notifying the user of the updated or added event information. This enables detailed analysis that reflects emotional information along with a summary of the meeting content, enabling quick and practical problem-solving in factory production management and maintenance.

[0299] "Collected meeting log data" refers to data recorded during a meeting, such as audio data, text chat, participant information, and the start and end times of the meeting.

[0300] "Extracting key points" refers to extracting specific keywords and phrases from the meeting log data and extracting important information to create a summary of the meeting content.

[0301] "Summary generation" refers to creating a short and concise summary of the entire meeting based on the extracted important points.

[0302] "Identifying progress" refers to understanding the progress of projects and tasks discussed in meetings and clarifying their status and degree of achievement.

[0303] "Identifying new issues" refers to identifying new problems or unresolved issues that arise during the meeting.

[0304] "Updating or adding events" refers to updating action items or tasks in your project management system or adding new events based on the progress of the meeting or new issues.

[0305] "Recognizing emotions from users' comments and attitudes" refers to detecting participants' emotional states by analyzing the audio and text of meeting log data.

[0306] "Reflecting emotional information in summaries" means including recognized emotional information in the summary content to provide detailed meeting content that takes into account changes in emotions and signs of stress.

[0307] "Notifying the user" refers to reporting updated or added event information, progress, new challenges, and emotional information to the user at an appropriate time.

[0308] This invention is a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support the efficient management of production meetings and maintenance meetings in factories. It also incorporates an emotion engine that recognizes the user's emotions. Below, we will explain how to implement this invention based on the roles of the server, terminal, and user.

[0309] 1. Collecting meeting log data

[0310] Users hold production meetings and maintenance meetings in the factory. All audio, text chat, and participant information from the meetings are recorded by the factory robot, which is equipped with a microphone, camera, and text chat collection tool.

[0311] 2. Acquiring meeting log data

[0312] After the meeting ends, the server retrieves the meeting log data collected from the factory robot. The retrieved data includes voice data, text data, a list of participants, and the start and end times. This data is then stored in a database.

[0313] 3. Summary and analysis of the meeting content

[0314] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy). The converted text data is then analyzed to extract key points. This process involves extracting specific keywords and important phrases from the verbatim audio data and generating a summary based on that. For example, the summary generated might say, "Project A is progressing well, but Team B is short on resources."

[0315] 4. Emotion Recognition by Emotion Engine

[0316] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's remarks and attitudes. It determines whether the user is feeling stressed based on the pitch of the voice data and the context of the text data. The recognized emotion information is reflected in the meeting summary, adding information such as, "The leader of Team B is feeling very stressed about the lack of resources."

[0317] 5. Event updates and additions

[0318] The server automatically updates and adds events to the project management system (e.g., JIRA) based on the analyzed progress, new issues, and sentiment information. For example, it adds the latest progress report to the "Project A Progress" event and generates an event for a new issue called "Lack of Resources."

[0319] 6. User Notification and Response

[0320] The device notifies the user of updated event information. Notifications include progress, new tasks, and emotional information. Notifications are sent at the most appropriate time, taking into account the user's state. For example, by notifying the user when they are relaxed, the device allows them to respond calmly to problems.

[0321] Specific examples

[0322] A regular meeting is held at a factory, and a factory robot records the meeting. After the meeting ends, the server obtains the data collected from the robot and converts it into text using natural language processing technology. It extracts important points and generates a summary such as, "Project A is progressing smoothly, but Team B's lack of resources is an issue." The server then uses an emotion engine to extract emotional information from the log data and adds information such as, "Team B members are feeling very stressed about the lack of resources." The terminal uses this information to update an event on the project management system (JIRA) and generates a new event for the issue, "Lack of Resources." It also sends a notification to the user, taking the emotional information into consideration. The user receives the notification, logs in to the project management system to check the details, considers countermeasures for the resource shortage, and initiates appropriate action.

[0323] Prompt Sentence Examples

[0324] Convert speech data to text, analyze the text for key points and sentiment, and use the results to update events in your project management system and add new issues.

[0325] As follows:

[0326] 1. Analyze the audio data and convert it into text.

[0327] 2. The converted text is analyzed using IBM Watson's sentiment analysis API.

[0328] 3. Based on the analysis results, add a new event to the project management system (JIRA) and send a notification.

[0329] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0330] Step 1:

[0331] Users hold production meetings and maintenance conferences in the factory. The robot uses a microphone, camera, and text chat collection tool to collect voice data, text chat data, participant information, and the start and end times of the conference.

[0332] Input: All meeting audio, text chat, and participant information

[0333] Output: Collected meeting log data

[0334] Step 2:

[0335] After the meeting ends, the server acquires the meeting log data collected from the robot.

[0336] Input: Meeting log data stored in the robot

[0337] Output: Meeting log data acquired by the server

[0338] Step 3:

[0339] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy), where a speech recognition model analyzes the audio data and compiles it into text.

[0340] Input: Meeting audio data

[0341] Output: Data converted to text

[0342] Step 4:

[0343] The server analyzes the text data and extracts key points, detecting specific keywords and phrases, and capturing key information to generate a summary of the meeting content.

[0344] Input: Data converted to text

[0345] Output: Extracted key points

[0346] Step 5:

[0347] The server generates a summary of the meeting content based on the extracted key points, using a generative AI model to concisely summarize the content of the summary you want to convey.

[0348] Input: Extracted key points

[0349] Output: Summary of the meeting

[0350] Step 6:

[0351] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's speech and behavior. It analyzes characteristics such as voice pitch and word choice to extract emotional information.

[0352] Input: Meeting log data (audio and text)

[0353] Output: Recognized emotion information

[0354] Step 7:

[0355] The server reflects the recognized emotional information in the summary of the meeting, and adds emotional information to the summary, allowing participants to understand the progress of the meeting and new issues from an emotional perspective.

[0356] Input: Summary of meeting content, recognized emotion information

[0357] Output: A summary of the meeting including emotional information

[0358] Step 8:

[0359] The server updates or adds events to the project management system (e.g., JIRA) based on the summary and sentiment information, so that progress and new issues are reflected in the project management system.

[0360] Input: A summary of the meeting including emotional information

[0361] Output: Updated or added event information

[0362] Step 9:

[0363] The device notifies the user of updated event information, including progress, new tasks, and emotion information, at appropriate times.

[0364] Input: Updated or added event information

[0365] Output: Notification sent to the user

[0366] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0367] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0368] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0369] [Second embodiment]

[0370] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0371] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0372] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0373] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0374] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0375] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0376] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0377] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0378] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0379] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0380] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0381] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0382] The present invention provides a system for supporting efficient management of conference contents, and a specific implementation method of the system based on the roles of the server, terminal, and user will be described below.

[0383] 1. Collecting meeting log data

[0384] When a user holds a meeting, the server automatically collects the meeting log data. For example, it uses the API of the online meeting platform to obtain the content of comments made during the meeting and participant information. This allows the server to collect meeting log data that includes information such as the meeting audio, text, participant list, start time, and end time.

[0385] 2. Summary and analysis of the meeting content

[0386] The server takes the collected meeting log data and converts it into text using natural language processing technology, then extracts key points from the text data and generates a concise summary.

[0387] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing well, but Team B is facing a lack of resources."

[0388] The server then uses this summary data to track progress and identify emerging issues, making it clear whether the project is progressing as planned or where problems lie.

[0389] 3. Event updates and additions

[0390] The terminal automatically updates events on the project management system based on the progress and task information received from the server, reflecting the progress status in existing events and adding new task-based events as necessary.

[0391] For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "Lack of resources."

[0392] 4. User Notification and Response

[0393] The device notifies the user of updated events and new challenges, allowing the user to quickly understand progress and take action to address any issues that arise.

[0394] For example, a user can receive a notification, log in to a project management system to view the details, and then consider how to address the resource shortage issue and take appropriate action.

[0395] Specific examples

[0396] Suppose a user holds an online project progress meeting. After the meeting ends, the server collects the meeting log data (audio, text, participant list, etc.) through the API of the online meeting platform.

[0397] The server analyzes the collected log data, generates a summary such as "Project A is progressing smoothly, but Team B is short on resources," and based on this summary, identifies the issue of "shortage of resources for Team B."

[0398] The terminal updates the "Project A Progress" event in the project management system and adds a new event called "Lack of resources."

[0399] The terminal notifies the user of these updates, and the user can log in and check the details. The user can take measures to address resource shortages and continue the smooth progress of the project.

[0400] In this way, the system based on the present invention realizes efficient project management through the summarization and analysis of meeting contents.

[0401] The processing flow will be explained below.

[0402] Step 1:

[0403] Users use the online meeting platform to start a project status meeting, during which all statements and participant information are recorded.

[0404] Step 2:

[0405] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, including audio data, text chat, participant list, start time, end time, etc.

[0406] Step 3:

[0407] The server stores the acquired meeting log data in a database, which makes it available for later analysis.

[0408] Step 4:

[0409] The server uses natural language processing technology to convert the voice data into text, which includes everything said during the meeting.

[0410] Step 5:

[0411] The server analyzes the text data and performs processing to extract key points, for example, identifying important keywords and phrases to use as the basis for summarizing.

[0412] Step 6:

[0413] The server generates a summary of the meeting based on the extracted key points, including progress reports, issues, and new topics.

[0414] Step 7:

[0415] The server then analyzes the generated summary to identify progress and new issues, for example, "Project A is progressing smoothly, but Team B is facing a resource shortage."

[0416] Step 8:

[0417] The terminal updates events on the project management system based on the progress status and new task information received from the server, adding progress information to existing events and generating events for new tasks.

[0418] Step 9:

[0419] The device notifies the user of updated event information, including a summary of progress and new tasks, keeping the user informed quickly and accurately.

[0420] Step 10:

[0421] Users receive notifications and log into the project management system to view details, and initiate appropriate action based on the progress and issue information provided.

[0422] These are the specific processing steps of the program. This system allows users to efficiently manage the progress of projects and quickly respond to new challenges.

[0423] Example 1

[0424] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0425] Conventional meeting management systems have made it difficult to effectively grasp important comments and issues raised during meetings and reflect them in subsequent project management. They also lacked a way to quickly notify users of meeting progress and new issues. This can lead to important information being overlooked during project progress, affecting the quality and progress of the final deliverables.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0427] In this invention, the server

[0428] A means of obtaining the collected meeting log data using the API of the online meeting platform;

[0429] means for converting the conference log data from voice data to text data using natural language processing technology;

[0430] means for extracting key points from the text data to generate summaries and identify progress and emerging issues;

[0431] A means for updating or adding events to a project management system based on the progress and new issues;

[0432] means for notifying a user of the updated or added event information;

[0433] This makes it possible to efficiently collect and analyze important meeting information and reflect it in the project management system. Furthermore, by promptly notifying users of progress and new issues, the project can be progressed appropriately and problems can be resolved.

[0434] "Conference log data" refers to information recorded during an online conference, such as speech content, voice data, text chat, participant list, and conference time.

[0435] An "online conference platform" is software or a service that allows multiple users to hold conferences over the Internet while sharing audio and video in real time.

[0436] "API" stands for Application Program Interface, an interface for sharing information and functions between different software systems.

[0437] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language, and is used to analyze text and voice data.

[0438] "Text data" is voice data converted into character information, and is used to facilitate analysis and display.

[0439] A "summary" is a short summary of important points extracted from the original text data.

[0440] "Progress" is information that indicates how far a project or the like has progressed relative to a plan.

[0441] An "issue" is a problem or obstacle that needs to be resolved as the project progresses.

[0442] A "project management system" is a system that centrally manages project planning, progress management, task allocation, etc., and supports efficient project management.

[0443] An "event" is an item that indicates information such as a specific progress or issue in the project management system.

[0444] "Notification" is a means by which a system notifies a user of specific information, and includes alerts and messages.

[0445] This invention is a system that supports efficient management of online meeting contents by effectively collecting and analyzing log data of online meetings and reflecting it in the related project management system. This system mainly operates based on the roles of the server, terminal, and user.

[0446] When a user starts a conference using the API of an online conference platform (e.g., an online conference application), the server automatically collects conference log data. This log data includes speech content (audio data), text chat, a participant list, the start and end times of the conference, etc. The audio data is processed to identify the speaker using speaker diarization technology.

[0447] After collecting the log data, the server stores it in storage and uses natural language processing technology to convert the voice data into text data. Specifically, it uses Google Cloud Speech-to-Text API or IBM Watson Speech to Text. The converted text data is then analyzed using Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4) to extract important points. The summary generated as a result of the analysis might include, for example, "Project A is progressing well, but Team B is facing a lack of resources." Based on this summary, the server identifies progress and new challenges.

[0448] The identified progress and issue information is sent from the server to the device. The device receives this information and automatically updates the event using the API of the project management system (e.g., project management software). Specifically, the new progress is reflected in the "Project A Progress" event, and an event based on the new issue, "Lack of Resources," is added. This process is performed through the device's API request, and the updated event information is saved in the project management system.

[0449] Updated and added event information is notified to the user via the terminal. The user can then log in to the project management system to check detailed information and take specific measures, such as allocating additional staff to resolve resource shortages.

[0450] Specific examples

[0451] Consider a case where a user holds an online project progress meeting. After the meeting ends, the server automatically collects the meeting log data (audio, text, participant list, meeting time, etc.) using the API of the online meeting platform. At this time, speaker diarization is applied to identify the speaker.

[0452] The server stores the collected log data in storage and converts the voice data into text using the Google Cloud Speech-to-Text API. The text data is then analyzed using OpenAI's GPT-4, which generates a summary that says, "Project A is progressing smoothly, but Team B's lack of resources is a problem," and identifies the "lack of resources" as a new issue.

[0453] The device receives the data sent from the server and uses the API of the project management software to update the "Project A Progress" event and add a new event called "Lack of Resources." For example, it sends an API request to add "Team B's resource shortage is an issue" to the event description.

[0454] The device notifies the user of these updates, who can then log into the project management software to view the details and take concrete action, such as allocating additional staff to address resource shortages.

[0455] Prompt Sentence Examples

[0456] Here is an example prompt for a generative AI model to explain the system in more detail:

[0457] "Please describe in detail a system that supports efficient management of meeting content. The system collects log data from online meetings and uses natural language processing techniques to summarize and identify issues. Based on this, the system updates events in a project management system and notifies users. Please also mention the specific APIs and analysis tools used."

[0458] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0459] Step 1:

[0460] When a user starts an online conference, the server uses the API of the online conference platform to obtain conference log data. The obtained data includes speech content (audio data), text chat, participant list, and conference start and end times. Specifically, this data is obtained through API calls and saved in internal storage. The input is the online conference session ID, and the output is temporary saved conference log data.

[0461] Step 2:

[0462] The server converts the acquired voice data into text data using natural language processing technology. It then uses the Google Cloud Speech-to-Text API to convert the voice data into text. The input is voice data, and the output is text data. Specifically, it sends voice data to the API and receives the returned text data.

[0463] Step 3:

[0464] The server analyzes the converted text data, extracts key points, and generates a summary using the Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4). The input is the text data, and the output is a summary, identified progress, and new challenges. Specifically, the server sends the text data to the analysis engine and receives the analysis results.

[0465] Step 4:

[0466] The server organizes the identified progress and new issues along with the summary, and sends the organized data to the device. The input is the analyzed summary data, and the output is notification data to the device. Specifically, the server sends the organized data to the device's API.

[0467] Step 5:

[0468] The device automatically updates the event using the API of the project management system based on the received data. It sends an API request to update the "Project A Progress" event with the latest progress information, and generates a new event based on the new issue of "Lack of resources." The input is progress information and issue information from the server, and the output is an updated event in the project management system. Specifically, it sends an API request to update the event details.

[0469] Step 6:

[0470] The terminal notifies the user of the updated event information. The user receives the notification and logs in to the project management system to check the detailed information. The input is the event information of the project management system, and the output is the notification to the user. The specific operation is to generate a notification message and send it to the user's device.

[0471] Step 7:

[0472] Upon receiving the notification, the user accesses the project management system and takes specific measures. For example, they may take action such as adding new staff to resolve resource shortages. The input is the notified event information, and the output is the implementation of specific measures. Specifically, the user logs in to the project management system, checks the details, and records the measures.

[0473] (Application example 1)

[0474] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] Currently, managing meeting content at manufacturing sites requires a lot of manual work, and recording and summarizing the content after meetings often takes a lot of time. This delays understanding progress and identifying new issues, making it difficult to respond quickly. Furthermore, with on-site work being hectic, there are limitations to accurately recording meeting content and quickly notifying relevant parties of the necessary information. To solve this situation, there is a need for a system that automatically collects and analyzes meeting logs and quickly notifies relevant parties of summaries and issues.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0477] In this invention, the server includes: means for acquiring collected meeting log data; means for extracting important points from the meeting log data and generating a summary; means for analyzing the summary and identifying progress and new issues; means for updating or adding events based on the progress and new issues; means for notifying the user of the updated or added event information; means installed in the smart glasses for recording the meeting log in real time; and means for automatically analyzing the meeting log to generate a summary and notifying the user of the progress and new issues. This improves the efficiency of meeting content management at factory manufacturing sites, enabling quick progress understanding and issue identification.

[0478] "Collected conference log data" refers to data that includes information such as audio recorded during a conference, text, a participant list, start time, and end time.

[0479] "Important points" refer to statements, issues, decisions, etc. that are considered particularly important within the content of the meeting.

[0480] A "summary" is a document or textual information that briefly summarizes what was discussed at a meeting.

[0481] "Progress" refers to the current state or stage of a project or task.

[0482] "New issues" refer to new problems or issues that need to be addressed that were identified during the meeting.

[0483] "Updating or adding an event" means modifying or creating new tasks or schedules in an existing project management system based on the latest information.

[0484] "Notifying users" means informing users of the generated update information and information about new issues via email, a notification system, or the like.

[0485] "Smart glasses" are a type of wearable device that refers to a glasses-type display device that has the ability to display and record information in real time.

[0486] "Real-time recording" means collecting audio and video simultaneously while the meeting is in progress and saving the data.

[0487] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[0488] The present invention relates to a system for efficiently managing meetings at a manufacturing site in a factory. The system is configured based on the roles of smart glasses, a server, a terminal, and a user.

[0489] Meeting log data collection

[0490] First, when a user holds a meeting, the smart glasses record the meeting log data in real time. The smart glasses are equipped with speech recognition and text conversion functions, and automatically collect speech data during the meeting and convert it into text data. This data is then sent to a server.

[0491] Meeting summary and analysis

[0492] The server receives the meeting log data sent from the smart glasses and converts it into text using natural language processing technology. It then extracts key points from the collected text data and generates a concise summary. The server uses the SpaCy model as its natural language processing technology. For example, it generates a summary such as, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0493] Event updates and additions

[0494] The terminal automatically updates events on the project management system based on the progress and issue information received from the server. It reflects the progress status in existing events and adds events based on new issues as necessary. For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "lack of resources."

[0495] User notification and response

[0496] The device notifies the user of updated events and new tasks, allowing the user to quickly understand the progress and take action to address any issues that arise. Specifically, email notifications are sent using Python's smtplib. For example, notifications are sent to email addresses such as "user1@example.com" and "user2@example.com."

[0497] Specific examples

[0498] For example, a regular meeting is held in the manufacturing department of a factory, and after the meeting, smart glasses collect the meeting log and generate a summary such as "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials." Based on this, a new issue is added to the project management system and the user is notified about it by email.

[0499] Prompt Sentence Examples

[0500] Generate a meeting log stating, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0501] This approach streamlines meeting content management on the factory floor, allowing for quick progress updates and the identification of emerging issues, while also enabling more advanced analysis and automation using generative AI models and prompts.

[0502] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0503] Step 1:

[0504] Collection and transmission of meeting log data

[0505] A user wears smart glasses and holds a conference. The glasses collect audio data during the conference in real time and convert it into text data. This text data includes the content of the speech, the speaker, a timestamp, etc. The smart glasses then send this data to a server.

[0506] Input: Audio data during the meeting

[0507] Output: Meeting log data in text format

[0508] Step 2:

[0509] Receiving and storing meeting log data

[0510] The server receives the meeting log data sent from the smart glasses and stores it in a database for later analysis.

[0511] Input: Text data sent from smart glasses

[0512] Output: Meeting log data stored in a database

[0513] Step 3:

[0514] Meeting summary generation

[0515] The server analyzes the saved meeting log data using natural language processing technology. It uses a generative AI model (e.g., SpaCy) to extract key points and generate a summary. For example, a summary might be generated that reads, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0516] Input: Meeting log data stored in a database

[0517] Output: Summary of meeting content

[0518] Step 4:

[0519] Identifying progress and new challenges

[0520] The server identifies progress and new issues from the generated summary, and uses natural language processing techniques to extract problem and progress information from keywords and phrases in the summary.

[0521] Input: Summary of meeting content

[0522] Output: Identified progress and challenges

[0523] Step 5:

[0524] Event updates and additions

[0525] The terminal automatically updates events in the project management system based on the identified progress and issue information. It reflects the progress in existing events and adds new events based on issues as needed. For example, it adds a new resource arrangement event based on the fact that "production line 2 is short of raw materials."

[0526] Input: Identified progress and challenges

[0527] Output: Updated or added event information

[0528] Step 6:

[0529] User Notification

[0530] The device notifies the user of updated events and new challenges via email or push notifications on the smartphone app, allowing the user to quickly understand progress and address any issues.

[0531] Input: Updated or added event information

[0532] Output: User notification

[0533] These steps allow for efficient management of meeting content and enable rapid response on-site.

[0534] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0535] This invention combines a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support efficient management of meeting content with an emotion engine that recognizes user emotions. Below, we will explain specific implementation methods of this invention based on the roles of the server, terminal, and user.

[0536] 1. Collecting meeting log data

[0537] Users use the online meeting platform to hold project progress meetings, and all statements, text chats, and participant information are recorded by the platform.

[0538] 2. Acquiring meeting log data

[0539] After the meeting ends, the server retrieves the meeting log data through the online meeting platform's API. The retrieved data includes audio data, text data, participant list, start time, and end time. The retrieved meeting log data is then stored in a database.

[0540] 3. Summary and analysis of the meeting content

[0541] The server uses natural language processing technology to convert the voice data into text, analyzes the converted text data, extracts important points, and generates a summary of the meeting content based on the extracted points.

[0542] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing smoothly, but Team B is experiencing a resource shortage."

[0543] 4. Emotion Recognition by Emotion Engine

[0544] During the analysis of the meeting log data, the server uses an emotion engine to recognize emotions from the user's remarks and attitudes. For example, it can determine whether the user is feeling stressed from the pitch of their voice and the choice of words.

[0545] The perceived emotional information is reflected in the meeting summary, and additional information is added, for example, in the form of "Team B's leader expresses strong concerns about the lack of resources."

[0546] 5. Event updates and additions

[0547] The terminal automatically updates and adds events to the project management system based on the progress, new issues, and emotion information sent from the server. For example, the terminal adds the latest progress report to the "Project A Progress" event and generates an event for the new issue of "Lack of resources."

[0548] 6. User Notification and Response

[0549] The device notifies the user of updated event information, including progress, new tasks, and user emotional information, at the most appropriate time.

[0550] For example, it is considered that by notifying the user when he or she is relaxed, the user can calmly deal with the problem.

[0551] Specific examples

[0552] Suppose a user holds an online project progress meeting. After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform.

[0553] The server analyzes the log data and converts it into text using natural language processing technology. It extracts key points and generates a summary such as, "Project A is progressing smoothly, but Team B is facing a lack of resources."

[0554] Furthermore, the server uses an emotion engine to extract emotional information from the log data and add information such as "Team B members are feeling very stressed about the lack of resources."

[0555] The device updates the events on the project management system based on the received information, generates a new event for the issue "resource shortage," and sends a notification to the user while taking into account emotional information.

[0556] Users receive notifications, log in to their project management systems to review the details, consider how to address the resource shortage, and initiate appropriate actions.

[0557] The present invention enables efficient management of meeting content including emotional information, and enables prompt and accurate responses to problems.

[0558] The processing flow will be explained below.

[0559] Step 1:

[0560] Users use an online meeting platform to initiate a project status meeting, during which all statements, text chats, and participant information are recorded.

[0561] Step 2:

[0562] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, which includes audio data, text data, a list of participants, and the start and end times.

[0563] Step 3:

[0564] The server stores the acquired log data in a database so that it can be used for later analysis.

[0565] Step 4:

[0566] The server uses natural language processing technology to convert the voice data into text, which includes all the content spoken during the meeting.

[0567] Step 5:

[0568] The server analyzes the converted text data, extracts important points and keywords from the meeting, and generates a summary based on the extracted points.

[0569] Step 6:

[0570] The server uses the summary to identify progress and emerging issues, such as "Project A is progressing well, but Team B is facing a lack of resources."

[0571] Step 7:

[0572] The server uses an emotion engine to recognize users' emotions from meeting log data, for example, determining whether they are stressed based on their voice pitch and word choice.

[0573] Step 8:

[0574] The server adds the recognized emotional information to the meeting summary, which includes emotional information such as "Team B members are highly stressed about the lack of resources."

[0575] Step 9:

[0576] The terminal automatically updates and adds events to the project management system based on the progress status, new tasks, and emotion information received from the server. Existing events are updated with the latest progress information, and new events are generated for new tasks.

[0577] Step 10:

[0578] The device notifies the user of updated event information, including progress, new tasks, and emotional information, and the content and timing of notifications are tailored to the user's emotional state.

[0579] Step 11:

[0580] Users receive notifications, log into their project management system to review details, and initiate appropriate action based on the progress, issue information, and sentiment information provided.

[0581] These are the specific processing steps based on the present invention, which combines an emotion engine. This system allows users to efficiently manage project progress and quickly respond to new challenges. In addition, taking emotion information into account promotes optimal behavior for problem solving.

[0582] Example 2

[0583] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] Conventional meeting management systems have difficulty efficiently grasping the progress and issues of meetings, and more accurate decision-making is required, especially by including participants' emotional information. Furthermore, since summarizing meeting content and extracting key points takes time and effort, automation is necessary. Furthermore, project management updates and notifications based on emotional information are also an important issue.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0586] In this invention, the server includes means for acquiring collected meeting log data, means for converting the meeting log data into voice data and text data, means for extracting important points from the voice data using natural language processing technology and generating a summary, means for extracting user emotional information from the text data and voice data using emotion analysis technology and incorporating the information into the summary, means for analyzing the summary to identify progress and new issues, means for updating or adding events to the project management system based on the progress and new issues, and means for notifying the user of the updated or added event information and emotional information. This enables efficient management of meeting content, accelerated decision-making including emotional information, and automatic updating and notification of the project management system.

[0587] "Conference log data" refers to data recorded during a conference, including statements made during the conference, text chat, participant information, and the start and end times of the conference.

[0588] "Audio data" refers to data recorded in audio file format of what was said during a conference.

[0589] "Text data" refers to text-format data converted from voice data, or text chat data recorded during a conference.

[0590] "Natural language processing technology" is a technology for analyzing language data and understanding its meaning, and in the present invention, it is a technology used as a means for converting voice data into text and extracting important points.

[0591] "Emotion analysis technology" is a technology that recognizes and extracts the emotional state of a speaker from text data and audio data.

[0592] The "summary" is a concise summary of the important points of the meeting content extracted using natural language processing technology.

[0593] "Progress" refers to the current state or progress of a project or task.

[0594] "Issues" are problems that arise during the project or areas that require improvement.

[0595] An "event" is an entry in a project management system that records and tracks actions on progress or new issues.

[0596] A "project management system" is a software system for planning, executing, and monitoring tasks or projects.

[0597] "Communication means" refers to the technology and methods by which servers and terminals exchange data with online meeting platforms and project management systems.

[0598] "Notification" is the act of notifying the user of updated event information or important messages.

[0599] The present invention provides a system for efficiently managing meeting log data collected using an online meeting platform and generating a detailed summary including user emotional information. In the following embodiments, specific implementation methods will be described based on the roles of the server, terminal, and user.

[0600] 1. Overview of the entire system

[0601] This system supports large-scale project management and efficient meeting management for remote teams. The system acquires meeting log data from an online meeting platform and analyzes it using natural language processing and sentiment analysis technologies. As a result, it is possible to grasp a summary of the meeting content, progress, new issues, and user sentiment. This automatically updates events in the project management system and sends notifications to users at optimal times.

[0602] 2. Hardware and Software Configuration

[0603] server

[0604] The server is a general-purpose computer with a powerful processor and sufficient memory, running the following software:

[0605] Operating System: Linux

[0606] Database: MySQL

[0607] Natural language processing technology: Google Cloud Speech-to-Text API, spaCy

[0608] Emotion analysis technology: Amazon Comprehend

[0609] Terminal

[0610] The terminal is a personal computer or mobile device operated by a user, running the following software:

[0611] Operating Systems: Windows, macOS, Android, iOS

[0612] Project management systems: JIRA, Asana

[0613] User

[0614] Users are project managers and meeting participants who participate in meetings through an online meeting platform. In particular, the following environment is required:

[0615] Online meeting platforms: Zoom, Microsoft Teams

[0616] 3. Specific examples of implementation

[0617] A user holds a project status meeting using Zoom. After the meeting ends, the server retrieves the meeting log data through the Zoom API. The server converts the acquired audio data into text using the Google Cloud Speech-to-Text API, analyzes the converted text data using natural language processing technology (spaCy), and extracts key points. The server then extracts sentiment information using Amazon Comprehend and combines the summary with the sentiment information.

[0618] Specific prompt examples:

[0619] "Generate a summary of a project status meeting. Analyze the following log data and include key points and sentiment information."

[0620] 4. Operational Flow

[0621] A user holds a meeting, and the meeting log data is recorded by the online meeting platform. After the meeting ends, the server retrieves the recorded data and performs natural language processing and sentiment analysis. A summary is generated based on the analysis results, and the event is updated and added to the project management system. The terminal notifies the user of the update, and the user can access the project management system to check the details.

[0622] The present invention enables efficient management of meeting content and quick and accurate responses that take into account user feelings, thereby increasing the success rate of projects.

[0623] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0624] Step 1: Hosting a meeting and collecting log data

[0625] A user uses an online conference platform to hold a project progress meeting. When the user starts the meeting, the online conference platform records the remarks made during the meeting, text chat, and participant information in real time. The input is the user's voice and text remarks, and the output is the meeting log data. This log data is stored on the server of the online conference platform.

[0626] Step 2: Retrieving meeting log data

[0627] After a meeting ends, the server retrieves the meeting log data through the API of the online meeting platform. Specifically, the server sends a request to the API endpoint to retrieve the meeting log data, including audio data, text data, a list of participants, and the start and end times. The input is the request to the API, and the output is the retrieved dataset. This dataset is then stored in a database.

[0628] Step 3: Transcribe the meeting content

[0629] The server converts the audio data into text using the Google Cloud Speech-to-Text API. The server sends the acquired audio data to the API and receives the corresponding text data. The input is audio data and the output is text data. This text data is saved as a temporary file and used in the next step.

[0630] Step 4: Analyze text data and generate summaries

[0631] The server uses an NLP (Natural Language Processing) library (e.g. spaCy) to analyze the converted text data and extract key points. The server analyzes the text data and extracts the main points of the meeting. The input is the text data, and the output is a list of key points. A summary is generated based on these key points. The server aggregates the extracted points and generates a summary of the meeting content. The output is the generated summary.

[0632] Step 5: Performing Emotion Recognition

[0633] The server sends text and audio data to the emotion engine Amazon Comprehend to extract the user's emotional information. The server passes the acquired data to the emotion engine and receives the analysis results. The input is text and audio data, and the output is emotional information. This emotional information is added to the summary. For example, information such as "The leader of Team B expresses strong concerns about the lack of resources" is added.

[0634] Step 6: Update your project management system

[0635] The device updates the project management system based on the progress, new tasks, and sentiment information sent from the server. The device receives data from the server and updates and adds events to the project management system through API. The input is the data received from the server, and the output is the updated event in the project management system. For example, a new report is added to "Project A Progress."

[0636] Step 7: Notify users

[0637] The device notifies the user of the updated event information. Based on the updated information, the device sends a notification to the user in an appropriate format (e.g., push notification, email, dashboard display). The input is the updated event information, and the output is a notification to the user. The user receives the notification and logs in to the project management system to check the detailed information. For example, specific instructions such as "Please start considering countermeasures for the resource shortage issue" are displayed.

[0638] The system enables efficient management of meeting content, generation of detailed summaries including emotional information, and automatic updates and notifications for project management systems.

[0639] (Application example 2)

[0640] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0641] In production management and maintenance meetings at factories, it is necessary to efficiently grasp progress and identify new issues. However, typical meeting recording systems are limited to summarizing meetings and extracting progress, and are unable to consider the emotions and stress levels of meeting participants. This poses a challenge, making it difficult to provide more detailed and practical problem-solving based on emotional information. In particular, in factories, where resource shortages and emergency responses are often required, a rapid response that takes into account the emotions of participants is required.

[0642] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0643] In this invention, the server includes means for acquiring collected meeting log data, means for extracting important points from the meeting log data and generating a summary, means for recognizing emotions from user comments and attitudes, means for reflecting the emotional information in the summary, means for updating or adding events based on the progress and new tasks, and means for notifying the user of the updated or added event information. This enables detailed analysis that reflects emotional information along with a summary of the meeting content, enabling quick and practical problem-solving in factory production management and maintenance.

[0644] "Collected meeting log data" refers to data recorded during a meeting, such as audio data, text chat, participant information, and the start and end times of the meeting.

[0645] "Extracting key points" refers to extracting specific keywords and phrases from the meeting log data and extracting important information to create a summary of the meeting content.

[0646] "Summary generation" refers to creating a short and concise summary of the entire meeting based on the extracted important points.

[0647] "Identifying progress" refers to understanding the progress of projects and tasks discussed in meetings and clarifying their status and degree of achievement.

[0648] "Identifying new issues" refers to identifying new problems or unresolved issues that arise during the meeting.

[0649] "Updating or adding events" refers to updating action items or tasks in your project management system or adding new events based on the progress of the meeting or new issues.

[0650] "Recognizing emotions from users' comments and attitudes" refers to detecting participants' emotional states by analyzing the audio and text of meeting log data.

[0651] "Reflecting emotional information in summaries" means including recognized emotional information in the summary content to provide detailed meeting content that takes into account changes in emotions and signs of stress.

[0652] "Notifying the user" refers to reporting updated or added event information, progress, new challenges, and emotional information to the user at an appropriate time.

[0653] This invention is a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support the efficient management of production meetings and maintenance meetings in factories. It also incorporates an emotion engine that recognizes the user's emotions. Below, we will explain how to implement this invention based on the roles of the server, terminal, and user.

[0654] 1. Collecting meeting log data

[0655] Users hold production meetings and maintenance meetings in the factory. All audio, text chat, and participant information from the meetings are recorded by the factory robot, which is equipped with a microphone, camera, and text chat collection tool.

[0656] 2. Acquiring meeting log data

[0657] After the meeting ends, the server retrieves the meeting log data collected from the factory robot. The retrieved data includes voice data, text data, a list of participants, and the start and end times. This data is then stored in a database.

[0658] 3. Summary and analysis of the meeting content

[0659] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy). The converted text data is then analyzed to extract key points. This process involves extracting specific keywords and important phrases from the verbatim audio data and generating a summary based on that. For example, the summary generated might say, "Project A is progressing well, but Team B is short on resources."

[0660] 4. Emotion Recognition by Emotion Engine

[0661] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's remarks and attitudes. It determines whether the user is feeling stressed based on the pitch of the voice data and the context of the text data. The recognized emotion information is reflected in the meeting summary, adding information such as, "The leader of Team B is feeling very stressed about the lack of resources."

[0662] 5. Event updates and additions

[0663] The server automatically updates and adds events to the project management system (e.g., JIRA) based on the analyzed progress, new issues, and sentiment information. For example, it adds the latest progress report to the "Project A Progress" event and generates an event for a new issue called "Lack of Resources."

[0664] 6. User Notification and Response

[0665] The device notifies the user of updated event information. Notifications include progress, new tasks, and emotional information. Notifications are sent at the most appropriate time, taking into account the user's state. For example, by notifying the user when they are relaxed, the device allows them to respond calmly to problems.

[0666] Specific examples

[0667] A regular meeting is held at a factory, and a factory robot records the meeting. After the meeting ends, the server obtains the data collected from the robot and converts it into text using natural language processing technology. It extracts important points and generates a summary such as, "Project A is progressing smoothly, but Team B's lack of resources is an issue." The server then uses an emotion engine to extract emotional information from the log data and adds information such as, "Team B members are feeling very stressed about the lack of resources." The terminal uses this information to update an event on the project management system (JIRA) and generates a new event for the issue, "Lack of Resources." It also sends a notification to the user, taking the emotional information into consideration. The user receives the notification, logs in to the project management system to check the details, considers countermeasures for the resource shortage, and initiates appropriate action.

[0668] Prompt Sentence Examples

[0669] Convert speech data to text, analyze the text for key points and sentiment, and use the results to update events in your project management system and add new issues.

[0670] As follows:

[0671] 1. Analyze the audio data and convert it into text.

[0672] 2. The converted text is analyzed using IBM Watson's sentiment analysis API.

[0673] 3. Based on the analysis results, add a new event to the project management system (JIRA) and send a notification.

[0674] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0675] Step 1:

[0676] Users hold production meetings and maintenance conferences in the factory. The robot uses a microphone, camera, and text chat collection tool to collect voice data, text chat data, participant information, and the start and end times of the conference.

[0677] Input: All meeting audio, text chat, and participant information

[0678] Output: Collected meeting log data

[0679] Step 2:

[0680] After the meeting ends, the server acquires the meeting log data collected from the robot.

[0681] Input: Meeting log data stored in the robot

[0682] Output: Meeting log data acquired by the server

[0683] Step 3:

[0684] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy), where a speech recognition model analyzes the audio data and compiles it into text.

[0685] Input: Meeting audio data

[0686] Output: Data converted to text

[0687] Step 4:

[0688] The server analyzes the text data and extracts key points, detecting specific keywords and phrases, and capturing key information to generate a summary of the meeting content.

[0689] Input: Data converted to text

[0690] Output: Extracted key points

[0691] Step 5:

[0692] The server generates a summary of the meeting content based on the extracted key points, using a generative AI model to concisely summarize the content of the summary you want to convey.

[0693] Input: Extracted key points

[0694] Output: Summary of the meeting

[0695] Step 6:

[0696] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's speech and behavior. It analyzes characteristics such as voice pitch and word choice to extract emotional information.

[0697] Input: Meeting log data (audio and text)

[0698] Output: Recognized emotion information

[0699] Step 7:

[0700] The server reflects the recognized emotional information in the summary of the meeting, and adds emotional information to the summary, allowing participants to understand the progress of the meeting and new issues from an emotional perspective.

[0701] Input: Summary of meeting content, recognized emotion information

[0702] Output: A summary of the meeting including emotional information

[0703] Step 8:

[0704] The server updates or adds events to the project management system (e.g., JIRA) based on the summary and sentiment information, so that progress and new issues are reflected in the project management system.

[0705] Input: A summary of the meeting including emotional information

[0706] Output: Updated or added event information

[0707] Step 9:

[0708] The device notifies the user of updated event information, including progress, new tasks, and emotion information, at appropriate times.

[0709] Input: Updated or added event information

[0710] Output: Notification sent to the user

[0711] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0712] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0713] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0714] [Third embodiment]

[0715] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0716] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0717] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0718] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0719] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0720] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0721] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0722] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0723] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0724] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0725] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0726] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0727] The present invention provides a system for supporting efficient management of conference contents, and a specific implementation method of the system based on the roles of the server, terminal, and user will be described below.

[0728] 1. Collecting meeting log data

[0729] When a user holds a meeting, the server automatically collects the meeting log data. For example, it uses the API of the online meeting platform to obtain the content of comments made during the meeting and participant information. This allows the server to collect meeting log data that includes information such as the meeting audio, text, participant list, start time, and end time.

[0730] 2. Summary and analysis of the meeting content

[0731] The server takes the collected meeting log data and converts it into text using natural language processing technology, then extracts key points from the text data and generates a concise summary.

[0732] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing well, but Team B is facing a lack of resources."

[0733] The server then uses this summary data to track progress and identify emerging issues, making it clear whether the project is progressing as planned or where problems lie.

[0734] 3. Event updates and additions

[0735] The terminal automatically updates events on the project management system based on the progress and task information received from the server, reflecting the progress status in existing events and adding new task-based events as necessary.

[0736] For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "Lack of resources."

[0737] 4. User Notification and Response

[0738] The device notifies the user of updated events and new challenges, allowing the user to quickly understand progress and take action to address any issues that arise.

[0739] For example, a user can receive a notification, log in to a project management system to view the details, and then consider how to address the resource shortage issue and take appropriate action.

[0740] Specific examples

[0741] Suppose a user holds an online project progress meeting. After the meeting ends, the server collects the meeting log data (audio, text, participant list, etc.) through the API of the online meeting platform.

[0742] The server analyzes the collected log data, generates a summary such as "Project A is progressing smoothly, but Team B is short on resources," and based on this summary, identifies the issue of "shortage of resources for Team B."

[0743] The terminal updates the "Project A Progress" event in the project management system and adds a new event called "Lack of resources."

[0744] The terminal notifies the user of these updates, and the user can log in and check the details. The user can take measures to address resource shortages and continue the smooth progress of the project.

[0745] In this way, the system based on the present invention realizes efficient project management through the summarization and analysis of meeting contents.

[0746] The processing flow will be explained below.

[0747] Step 1:

[0748] Users use the online meeting platform to start a project status meeting, during which all statements and participant information are recorded.

[0749] Step 2:

[0750] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, including audio data, text chat, participant list, start time, end time, etc.

[0751] Step 3:

[0752] The server stores the acquired meeting log data in a database, which makes it available for later analysis.

[0753] Step 4:

[0754] The server uses natural language processing technology to convert the voice data into text, which includes everything said during the meeting.

[0755] Step 5:

[0756] The server analyzes the text data and performs processing to extract key points, for example, identifying important keywords and phrases to use as the basis for summarizing.

[0757] Step 6:

[0758] The server generates a summary of the meeting based on the extracted key points, including progress reports, issues, and new topics.

[0759] Step 7:

[0760] The server then analyzes the generated summary to identify progress and new issues, for example, "Project A is progressing smoothly, but Team B is facing a resource shortage."

[0761] Step 8:

[0762] The terminal updates events on the project management system based on the progress status and new task information received from the server, adding progress information to existing events and generating events for new tasks.

[0763] Step 9:

[0764] The device notifies the user of updated event information, including a summary of progress and new tasks, keeping the user informed quickly and accurately.

[0765] Step 10:

[0766] Users receive notifications and log into the project management system to view details, and initiate appropriate action based on the progress and issue information provided.

[0767] These are the specific processing steps of the program. This system allows users to efficiently manage the progress of projects and quickly respond to new challenges.

[0768] Example 1

[0769] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0770] Conventional meeting management systems have made it difficult to effectively grasp important comments and issues raised during meetings and reflect them in subsequent project management. They also lacked a way to quickly notify users of meeting progress and new issues. This can lead to important information being overlooked during project progress, affecting the quality and progress of the final deliverables.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0772] In this invention, the server

[0773] A means of obtaining the collected meeting log data using the API of the online meeting platform;

[0774] means for converting the conference log data from voice data to text data using natural language processing technology;

[0775] means for extracting key points from the text data to generate summaries and identify progress and emerging issues;

[0776] A means for updating or adding events to a project management system based on the progress and new issues;

[0777] means for notifying a user of the updated or added event information;

[0778] This makes it possible to efficiently collect and analyze important meeting information and reflect it in the project management system. Furthermore, by promptly notifying users of progress and new issues, the project can be progressed appropriately and problems can be resolved.

[0779] "Conference log data" refers to information recorded during an online conference, such as speech content, voice data, text chat, participant list, and conference time.

[0780] An "online conference platform" is software or a service that allows multiple users to hold conferences over the Internet while sharing audio and video in real time.

[0781] "API" stands for Application Program Interface, an interface for sharing information and functions between different software systems.

[0782] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language, and is used to analyze text and voice data.

[0783] "Text data" is voice data converted into character information, and is used to facilitate analysis and display.

[0784] A "summary" is a short summary of important points extracted from the original text data.

[0785] "Progress" is information that indicates how far a project or the like has progressed relative to a plan.

[0786] An "issue" is a problem or obstacle that needs to be resolved as the project progresses.

[0787] A "project management system" is a system that centrally manages project planning, progress management, task allocation, etc., and supports efficient project management.

[0788] An "event" is an item that indicates information such as a specific progress or issue in the project management system.

[0789] "Notification" is a means by which a system notifies a user of specific information, and includes alerts and messages.

[0790] This invention is a system that supports efficient management of online meeting contents by effectively collecting and analyzing log data of online meetings and reflecting it in the related project management system. This system mainly operates based on the roles of the server, terminal, and user.

[0791] When a user starts a conference using the API of an online conference platform (e.g., an online conference application), the server automatically collects conference log data. This log data includes speech content (audio data), text chat, a participant list, the start and end times of the conference, etc. The audio data is processed to identify the speaker using speaker diarization technology.

[0792] After collecting the log data, the server stores it in storage and uses natural language processing technology to convert the voice data into text data. Specifically, it uses Google Cloud Speech-to-Text API or IBM Watson Speech to Text. The converted text data is then analyzed using Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4) to extract important points. The summary generated as a result of the analysis might include, for example, "Project A is progressing well, but Team B is facing a lack of resources." Based on this summary, the server identifies progress and new challenges.

[0793] The identified progress and issue information is sent from the server to the device. The device receives this information and automatically updates the event using the API of the project management system (e.g., project management software). Specifically, the new progress is reflected in the "Project A Progress" event, and an event based on the new issue, "Lack of Resources," is added. This process is performed through the device's API request, and the updated event information is saved in the project management system.

[0794] Updated and added event information is notified to the user via the terminal. The user can then log in to the project management system to check detailed information and take specific measures, such as allocating additional staff to resolve resource shortages.

[0795] Specific examples

[0796] Consider a case where a user holds an online project progress meeting. After the meeting ends, the server automatically collects the meeting log data (audio, text, participant list, meeting time, etc.) using the API of the online meeting platform. At this time, speaker diarization is applied to identify the speaker.

[0797] The server stores the collected log data in storage and converts the voice data into text using the Google Cloud Speech-to-Text API. The text data is then analyzed using OpenAI's GPT-4, which generates a summary that says, "Project A is progressing smoothly, but Team B's lack of resources is a problem," and identifies the "lack of resources" as a new issue.

[0798] The device receives the data sent from the server and uses the API of the project management software to update the "Project A Progress" event and add a new event called "Lack of Resources." For example, it sends an API request to add "Team B's resource shortage is an issue" to the event description.

[0799] The device notifies the user of these updates, who can then log into the project management software to view the details and take concrete action, such as allocating additional staff to address resource shortages.

[0800] Prompt Sentence Examples

[0801] Here is an example prompt for a generative AI model to explain the system in more detail:

[0802] "Please describe in detail a system that supports efficient management of meeting content. The system collects log data from online meetings and uses natural language processing techniques to summarize and identify issues. Based on this, the system updates events in a project management system and notifies users. Please also mention the specific APIs and analysis tools used."

[0803] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0804] Step 1:

[0805] When a user starts an online conference, the server uses the API of the online conference platform to obtain conference log data. The obtained data includes speech content (audio data), text chat, participant list, and conference start and end times. Specifically, this data is obtained through API calls and saved in internal storage. The input is the online conference session ID, and the output is temporary saved conference log data.

[0806] Step 2:

[0807] The server converts the acquired voice data into text data using natural language processing technology. It then uses the Google Cloud Speech-to-Text API to convert the voice data into text. The input is voice data, and the output is text data. Specifically, it sends voice data to the API and receives the returned text data.

[0808] Step 3:

[0809] The server analyzes the converted text data, extracts key points, and generates a summary using the Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4). The input is the text data, and the output is a summary, identified progress, and new challenges. Specifically, the server sends the text data to the analysis engine and receives the analysis results.

[0810] Step 4:

[0811] The server organizes the identified progress and new issues along with the summary, and sends the organized data to the device. The input is the analyzed summary data, and the output is notification data to the device. Specifically, the server sends the organized data to the device's API.

[0812] Step 5:

[0813] The device automatically updates the event using the API of the project management system based on the received data. It sends an API request to update the "Project A Progress" event with the latest progress information, and generates a new event based on the new issue of "Lack of resources." The input is progress information and issue information from the server, and the output is an updated event in the project management system. Specifically, it sends an API request to update the event details.

[0814] Step 6:

[0815] The terminal notifies the user of the updated event information. The user receives the notification and logs in to the project management system to check the detailed information. The input is the event information of the project management system, and the output is the notification to the user. The specific operation is to generate a notification message and send it to the user's device.

[0816] Step 7:

[0817] Upon receiving the notification, the user accesses the project management system and takes specific measures. For example, they may take action such as adding new staff to resolve resource shortages. The input is the notified event information, and the output is the implementation of specific measures. Specifically, the user logs in to the project management system, checks the details, and records the measures.

[0818] (Application example 1)

[0819] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0820] Currently, managing meeting content at manufacturing sites requires a lot of manual work, and recording and summarizing the content after meetings often takes a lot of time. This delays understanding progress and identifying new issues, making it difficult to respond quickly. Furthermore, with on-site work being hectic, there are limitations to accurately recording meeting content and quickly notifying relevant parties of the necessary information. To solve this situation, there is a need for a system that automatically collects and analyzes meeting logs and quickly notifies relevant parties of summaries and issues.

[0821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0822] In this invention, the server includes: means for acquiring collected meeting log data; means for extracting important points from the meeting log data and generating a summary; means for analyzing the summary and identifying progress and new issues; means for updating or adding events based on the progress and new issues; means for notifying the user of the updated or added event information; means installed in the smart glasses for recording the meeting log in real time; and means for automatically analyzing the meeting log to generate a summary and notifying the user of the progress and new issues. This improves the efficiency of meeting content management at factory manufacturing sites, enabling quick progress understanding and issue identification.

[0823] "Collected conference log data" refers to data that includes information such as audio recorded during a conference, text, a participant list, start time, and end time.

[0824] "Important points" refer to statements, issues, decisions, etc. that are considered particularly important within the content of the meeting.

[0825] A "summary" is a document or textual information that briefly summarizes what was discussed at a meeting.

[0826] "Progress" refers to the current state or stage of a project or task.

[0827] "New issues" refer to new problems or issues that need to be addressed that were identified during the meeting.

[0828] "Updating or adding an event" means modifying or creating new tasks or schedules in an existing project management system based on the latest information.

[0829] "Notifying users" means informing users of the generated update information and information about new issues via email, a notification system, or the like.

[0830] "Smart glasses" are a type of wearable device that refers to a glasses-type display device that has the ability to display and record information in real time.

[0831] "Real-time recording" means collecting audio and video simultaneously while the meeting is in progress and saving the data.

[0832] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[0833] The present invention relates to a system for efficiently managing meetings at a manufacturing site in a factory. The system is configured based on the roles of smart glasses, a server, a terminal, and a user.

[0834] Meeting log data collection

[0835] First, when a user holds a meeting, the smart glasses record the meeting log data in real time. The smart glasses are equipped with speech recognition and text conversion functions, and automatically collect speech data during the meeting and convert it into text data. This data is then sent to a server.

[0836] Meeting summary and analysis

[0837] The server receives the meeting log data sent from the smart glasses and converts it into text using natural language processing technology. It then extracts key points from the collected text data and generates a concise summary. The server uses the SpaCy model as its natural language processing technology. For example, it generates a summary such as, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0838] Event updates and additions

[0839] The terminal automatically updates events on the project management system based on the progress and issue information received from the server. It reflects the progress status in existing events and adds events based on new issues as necessary. For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "lack of resources."

[0840] User notification and response

[0841] The device notifies the user of updated events and new tasks, allowing the user to quickly understand the progress and take action to address any issues that arise. Specifically, email notifications are sent using Python's smtplib. For example, notifications are sent to email addresses such as "user1@example.com" and "user2@example.com."

[0842] Specific examples

[0843] For example, a regular meeting is held in the manufacturing department of a factory, and after the meeting, smart glasses collect the meeting log and generate a summary such as "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials." Based on this, a new issue is added to the project management system and the user is notified about it by email.

[0844] Prompt Sentence Examples

[0845] Generate a meeting log stating, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0846] This approach streamlines meeting content management on the factory floor, allowing for quick progress updates and the identification of emerging issues, while also enabling more advanced analysis and automation using generative AI models and prompts.

[0847] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0848] Step 1:

[0849] Collection and transmission of meeting log data

[0850] A user wears smart glasses and holds a conference. The glasses collect audio data during the conference in real time and convert it into text data. This text data includes the content of the speech, the speaker, a timestamp, etc. The smart glasses then send this data to a server.

[0851] Input: Audio data during the meeting

[0852] Output: Meeting log data in text format

[0853] Step 2:

[0854] Receiving and storing meeting log data

[0855] The server receives the meeting log data sent from the smart glasses and stores it in a database for later analysis.

[0856] Input: Text data sent from smart glasses

[0857] Output: Meeting log data stored in a database

[0858] Step 3:

[0859] Meeting summary generation

[0860] The server analyzes the saved meeting log data using natural language processing technology. It uses a generative AI model (e.g., SpaCy) to extract key points and generate a summary. For example, a summary might be generated that reads, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[0861] Input: Meeting log data stored in a database

[0862] Output: Summary of meeting content

[0863] Step 4:

[0864] Identifying progress and new challenges

[0865] The server identifies progress and new issues from the generated summary, and uses natural language processing techniques to extract problem and progress information from keywords and phrases in the summary.

[0866] Input: Summary of meeting content

[0867] Output: Identified progress and challenges

[0868] Step 5:

[0869] Event updates and additions

[0870] The terminal automatically updates events in the project management system based on the identified progress and issue information. It reflects the progress in existing events and adds new events based on issues as needed. For example, it adds a new resource arrangement event based on the fact that "production line 2 is short of raw materials."

[0871] Input: Identified progress and challenges

[0872] Output: Updated or added event information

[0873] Step 6:

[0874] User Notification

[0875] The device notifies the user of updated events and new challenges via email or push notifications on the smartphone app, allowing the user to quickly understand progress and address any issues.

[0876] Input: Updated or added event information

[0877] Output: User notification

[0878] These steps allow for efficient management of meeting content and enable rapid response on-site.

[0879] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0880] This invention combines a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support efficient management of meeting content with an emotion engine that recognizes user emotions. Below, we will explain specific implementation methods of this invention based on the roles of the server, terminal, and user.

[0881] 1. Collecting meeting log data

[0882] Users use the online meeting platform to hold project progress meetings, and all statements, text chats, and participant information are recorded by the platform.

[0883] 2. Acquiring meeting log data

[0884] After the meeting ends, the server retrieves the meeting log data through the online meeting platform's API. The retrieved data includes audio data, text data, participant list, start time, and end time. The retrieved meeting log data is then stored in a database.

[0885] 3. Summary and analysis of the meeting content

[0886] The server uses natural language processing technology to convert the voice data into text, analyzes the converted text data, extracts important points, and generates a summary of the meeting content based on the extracted points.

[0887] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing smoothly, but Team B is experiencing a resource shortage."

[0888] 4. Emotion Recognition by Emotion Engine

[0889] During the analysis of the meeting log data, the server uses an emotion engine to recognize emotions from the user's remarks and attitudes. For example, it can determine whether the user is feeling stressed from the pitch of their voice and the choice of words.

[0890] The perceived emotional information is reflected in the meeting summary, and additional information is added, for example, in the form of "Team B's leader expresses strong concerns about the lack of resources."

[0891] 5. Event updates and additions

[0892] The terminal automatically updates and adds events to the project management system based on the progress, new issues, and emotion information sent from the server. For example, the terminal adds the latest progress report to the "Project A Progress" event and generates an event for the new issue of "Lack of resources."

[0893] 6. User Notification and Response

[0894] The device notifies the user of updated event information, including progress, new tasks, and user emotional information, at the most appropriate time.

[0895] For example, it is considered that by notifying the user when he or she is relaxed, the user can calmly deal with the problem.

[0896] Specific examples

[0897] Suppose a user holds an online project progress meeting. After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform.

[0898] The server analyzes the log data and converts it into text using natural language processing technology. It extracts key points and generates a summary such as, "Project A is progressing smoothly, but Team B is facing a lack of resources."

[0899] Furthermore, the server uses an emotion engine to extract emotional information from the log data and add information such as "Team B members are feeling very stressed about the lack of resources."

[0900] The device updates the events on the project management system based on the received information, generates a new event for the issue "resource shortage," and sends a notification to the user while taking into account emotional information.

[0901] Users receive notifications, log in to their project management systems to review the details, consider how to address the resource shortage, and initiate appropriate actions.

[0902] The present invention enables efficient management of meeting content including emotional information, and enables prompt and accurate responses to problems.

[0903] The processing flow will be explained below.

[0904] Step 1:

[0905] Users use an online meeting platform to initiate a project status meeting, during which all statements, text chats, and participant information are recorded.

[0906] Step 2:

[0907] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, which includes audio data, text data, a list of participants, and the start and end times.

[0908] Step 3:

[0909] The server stores the acquired log data in a database so that it can be used for later analysis.

[0910] Step 4:

[0911] The server uses natural language processing technology to convert the voice data into text, which includes all the content spoken during the meeting.

[0912] Step 5:

[0913] The server analyzes the converted text data, extracts important points and keywords from the meeting, and generates a summary based on the extracted points.

[0914] Step 6:

[0915] The server uses the summary to identify progress and emerging issues, such as "Project A is progressing well, but Team B is facing a lack of resources."

[0916] Step 7:

[0917] The server uses an emotion engine to recognize users' emotions from meeting log data, for example, determining whether they are stressed based on their voice pitch and word choice.

[0918] Step 8:

[0919] The server adds the recognized emotional information to the meeting summary, which includes emotional information such as "Team B members are highly stressed about the lack of resources."

[0920] Step 9:

[0921] The terminal automatically updates and adds events to the project management system based on the progress status, new tasks, and emotion information received from the server. Existing events are updated with the latest progress information, and new events are generated for new tasks.

[0922] Step 10:

[0923] The device notifies the user of updated event information, including progress, new tasks, and emotional information, and the content and timing of notifications are tailored to the user's emotional state.

[0924] Step 11:

[0925] Users receive notifications, log into their project management system to review details, and initiate appropriate action based on the progress, issue information, and sentiment information provided.

[0926] These are the specific processing steps based on the present invention, which combines an emotion engine. This system allows users to efficiently manage project progress and quickly respond to new challenges. In addition, taking emotion information into account promotes optimal behavior for problem solving.

[0927] Example 2

[0928] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0929] Conventional meeting management systems have difficulty efficiently grasping the progress and issues of meetings, and more accurate decision-making is required, especially by including participants' emotional information. Furthermore, since summarizing meeting content and extracting key points takes time and effort, automation is necessary. Furthermore, project management updates and notifications based on emotional information are also an important issue.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0931] In this invention, the server includes means for acquiring collected meeting log data, means for converting the meeting log data into voice data and text data, means for extracting important points from the voice data using natural language processing technology and generating a summary, means for extracting user emotional information from the text data and voice data using emotion analysis technology and incorporating the information into the summary, means for analyzing the summary to identify progress and new issues, means for updating or adding events to the project management system based on the progress and new issues, and means for notifying the user of the updated or added event information and emotional information. This enables efficient management of meeting content, accelerated decision-making including emotional information, and automatic updating and notification of the project management system.

[0932] "Conference log data" refers to data recorded during a conference, including statements made during the conference, text chat, participant information, and the start and end times of the conference.

[0933] "Audio data" refers to data recorded in audio file format of what was said during a conference.

[0934] "Text data" refers to text-format data converted from voice data, or text chat data recorded during a conference.

[0935] "Natural language processing technology" is a technology for analyzing language data and understanding its meaning, and in the present invention, it is a technology used as a means for converting voice data into text and extracting important points.

[0936] "Emotion analysis technology" is a technology that recognizes and extracts the emotional state of a speaker from text data and audio data.

[0937] The "summary" is a concise summary of the important points of the meeting content extracted using natural language processing technology.

[0938] "Progress" refers to the current state or progress of a project or task.

[0939] "Issues" are problems that arise during the project or areas that require improvement.

[0940] An "event" is an entry in a project management system that records and tracks actions on progress or new issues.

[0941] A "project management system" is a software system for planning, executing, and monitoring tasks or projects.

[0942] "Communication means" refers to the technology and methods by which servers and terminals exchange data with online meeting platforms and project management systems.

[0943] "Notification" is the act of notifying the user of updated event information or important messages.

[0944] The present invention provides a system for efficiently managing meeting log data collected using an online meeting platform and generating a detailed summary including user emotional information. In the following embodiments, specific implementation methods will be described based on the roles of the server, terminal, and user.

[0945] 1. Overview of the entire system

[0946] This system supports large-scale project management and efficient meeting management for remote teams. The system acquires meeting log data from an online meeting platform and analyzes it using natural language processing and sentiment analysis technologies. As a result, it is possible to grasp a summary of the meeting content, progress, new issues, and user sentiment. This automatically updates events in the project management system and sends notifications to users at optimal times.

[0947] 2. Hardware and Software Configuration

[0948] server

[0949] The server is a general-purpose computer with a powerful processor and sufficient memory, running the following software:

[0950] Operating System: Linux

[0951] Database: MySQL

[0952] Natural language processing technology: Google Cloud Speech-to-Text API, spaCy

[0953] Emotion analysis technology: Amazon Comprehend

[0954] Terminal

[0955] The terminal is a personal computer or mobile device operated by a user, running the following software:

[0956] Operating Systems: Windows, macOS, Android, iOS

[0957] Project management systems: JIRA, Asana

[0958] User

[0959] Users are project managers and meeting participants who participate in meetings through an online meeting platform. In particular, the following environment is required:

[0960] Online meeting platforms: Zoom, Microsoft Teams

[0961] 3. Specific examples of implementation

[0962] A user holds a project status meeting using Zoom. After the meeting ends, the server retrieves the meeting log data through the Zoom API. The server converts the acquired audio data into text using the Google Cloud Speech-to-Text API, analyzes the converted text data using natural language processing technology (spaCy), and extracts key points. The server then extracts sentiment information using Amazon Comprehend and combines the summary with the sentiment information.

[0963] Specific prompt examples:

[0964] "Generate a summary of a project status meeting. Analyze the following log data and include key points and sentiment information."

[0965] 4. Operational Flow

[0966] A user holds a meeting, and the meeting log data is recorded by the online meeting platform. After the meeting ends, the server retrieves the recorded data and performs natural language processing and sentiment analysis. A summary is generated based on the analysis results, and the event is updated and added to the project management system. The terminal notifies the user of the update, and the user can access the project management system to check the details.

[0967] The present invention enables efficient management of meeting content and quick and accurate responses that take into account user feelings, thereby increasing the success rate of projects.

[0968] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0969] Step 1: Hosting a meeting and collecting log data

[0970] A user uses an online conference platform to hold a project progress meeting. When the user starts the meeting, the online conference platform records the remarks made during the meeting, text chat, and participant information in real time. The input is the user's voice and text remarks, and the output is the meeting log data. This log data is stored on the server of the online conference platform.

[0971] Step 2: Retrieving meeting log data

[0972] After a meeting ends, the server retrieves the meeting log data through the API of the online meeting platform. Specifically, the server sends a request to the API endpoint to retrieve the meeting log data, including audio data, text data, a list of participants, and the start and end times. The input is the request to the API, and the output is the retrieved dataset. This dataset is then stored in a database.

[0973] Step 3: Transcribe the meeting content

[0974] The server converts the audio data into text using the Google Cloud Speech-to-Text API. The server sends the acquired audio data to the API and receives the corresponding text data. The input is audio data and the output is text data. This text data is saved as a temporary file and used in the next step.

[0975] Step 4: Analyze text data and generate summaries

[0976] The server uses an NLP (Natural Language Processing) library (e.g. spaCy) to analyze the converted text data and extract key points. The server analyzes the text data and extracts the main points of the meeting. The input is the text data, and the output is a list of key points. A summary is generated based on these key points. The server aggregates the extracted points and generates a summary of the meeting content. The output is the generated summary.

[0977] Step 5: Performing Emotion Recognition

[0978] The server sends text and audio data to the emotion engine Amazon Comprehend to extract the user's emotional information. The server passes the acquired data to the emotion engine and receives the analysis results. The input is text and audio data, and the output is emotional information. This emotional information is added to the summary. For example, information such as "The leader of Team B expresses strong concerns about the lack of resources" is added.

[0979] Step 6: Update your project management system

[0980] The device updates the project management system based on the progress, new tasks, and sentiment information sent from the server. The device receives data from the server and updates and adds events to the project management system through API. The input is the data received from the server, and the output is the updated event in the project management system. For example, a new report is added to "Project A Progress."

[0981] Step 7: Notify users

[0982] The device notifies the user of the updated event information. Based on the updated information, the device sends a notification to the user in an appropriate format (e.g., push notification, email, dashboard display). The input is the updated event information, and the output is a notification to the user. The user receives the notification and logs in to the project management system to check the detailed information. For example, specific instructions such as "Please start considering countermeasures for the resource shortage issue" are displayed.

[0983] The system enables efficient management of meeting content, generation of detailed summaries including emotional information, and automatic updates and notifications for project management systems.

[0984] (Application example 2)

[0985] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0986] In production management and maintenance meetings at factories, it is necessary to efficiently grasp progress and identify new issues. However, typical meeting recording systems are limited to summarizing meetings and extracting progress, and are unable to consider the emotions and stress levels of meeting participants. This poses a challenge, making it difficult to provide more detailed and practical problem-solving based on emotional information. In particular, in factories, where resource shortages and emergency responses are often required, a rapid response that takes into account the emotions of participants is required.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0988] In this invention, the server includes means for acquiring collected meeting log data, means for extracting important points from the meeting log data and generating a summary, means for recognizing emotions from user comments and attitudes, means for reflecting the emotional information in the summary, means for updating or adding events based on the progress and new tasks, and means for notifying the user of the updated or added event information. This enables detailed analysis that reflects emotional information along with a summary of the meeting content, enabling quick and practical problem-solving in factory production management and maintenance.

[0989] "Collected meeting log data" refers to data recorded during a meeting, such as audio data, text chat, participant information, and the start and end times of the meeting.

[0990] "Extracting key points" refers to extracting specific keywords and phrases from the meeting log data and extracting important information to create a summary of the meeting content.

[0991] "Summary generation" refers to creating a short and concise summary of the entire meeting based on the extracted important points.

[0992] "Identifying progress" refers to understanding the progress of projects and tasks discussed in meetings and clarifying their status and degree of achievement.

[0993] "Identifying new issues" refers to identifying new problems or unresolved issues that arise during the meeting.

[0994] "Updating or adding events" refers to updating action items or tasks in your project management system or adding new events based on the progress of the meeting or new issues.

[0995] "Recognizing emotions from users' comments and attitudes" refers to detecting participants' emotional states by analyzing the audio and text of meeting log data.

[0996] "Reflecting emotional information in summaries" means including recognized emotional information in the summary content to provide detailed meeting content that takes into account changes in emotions and signs of stress.

[0997] "Notifying the user" refers to reporting updated or added event information, progress, new challenges, and emotional information to the user at an appropriate time.

[0998] This invention is a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support the efficient management of production meetings and maintenance meetings in factories. It also incorporates an emotion engine that recognizes the user's emotions. Below, we will explain how to implement this invention based on the roles of the server, terminal, and user.

[0999] 1. Collecting meeting log data

[1000] Users hold production meetings and maintenance meetings in the factory. All audio, text chat, and participant information from the meetings are recorded by the factory robot, which is equipped with a microphone, camera, and text chat collection tool.

[1001] 2. Acquiring meeting log data

[1002] After the meeting ends, the server retrieves the meeting log data collected from the factory robot. The retrieved data includes voice data, text data, a list of participants, and the start and end times. This data is then stored in a database.

[1003] 3. Summary and analysis of the meeting content

[1004] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy). The converted text data is then analyzed to extract key points. This process involves extracting specific keywords and important phrases from the verbatim audio data and generating a summary based on that. For example, the summary generated might say, "Project A is progressing well, but Team B is short on resources."

[1005] 4. Emotion Recognition by Emotion Engine

[1006] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's remarks and attitudes. It determines whether the user is feeling stressed based on the pitch of the voice data and the context of the text data. The recognized emotion information is reflected in the meeting summary, adding information such as, "The leader of Team B is feeling very stressed about the lack of resources."

[1007] 5. Event updates and additions

[1008] The server automatically updates and adds events to the project management system (e.g., JIRA) based on the analyzed progress, new issues, and sentiment information. For example, it adds the latest progress report to the "Project A Progress" event and generates an event for a new issue called "Lack of Resources."

[1009] 6. User Notification and Response

[1010] The device notifies the user of updated event information. Notifications include progress, new tasks, and emotional information. Notifications are sent at the most appropriate time, taking into account the user's state. For example, by notifying the user when they are relaxed, the device allows them to respond calmly to problems.

[1011] Specific examples

[1012] A regular meeting is held at a factory, and a factory robot records the meeting. After the meeting ends, the server obtains the data collected from the robot and converts it into text using natural language processing technology. It extracts important points and generates a summary such as, "Project A is progressing smoothly, but Team B's lack of resources is an issue." The server then uses an emotion engine to extract emotional information from the log data and adds information such as, "Team B members are feeling very stressed about the lack of resources." The terminal uses this information to update an event on the project management system (JIRA) and generates a new event for the issue, "Lack of Resources." It also sends a notification to the user, taking the emotional information into consideration. The user receives the notification, logs in to the project management system to check the details, considers countermeasures for the resource shortage, and initiates appropriate action.

[1013] Prompt Sentence Examples

[1014] Convert speech data to text, analyze the text for key points and sentiment, and use the results to update events in your project management system and add new issues.

[1015] As follows:

[1016] 1. Analyze the audio data and convert it into text.

[1017] 2. The converted text is analyzed using IBM Watson's sentiment analysis API.

[1018] 3. Based on the analysis results, add a new event to the project management system (JIRA) and send a notification.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] Users hold production meetings and maintenance conferences in the factory. The robot uses a microphone, camera, and text chat collection tool to collect voice data, text chat data, participant information, and the start and end times of the conference.

[1022] Input: All meeting audio, text chat, and participant information

[1023] Output: Collected meeting log data

[1024] Step 2:

[1025] After the meeting ends, the server acquires the meeting log data collected from the robot.

[1026] Input: Meeting log data stored in the robot

[1027] Output: Meeting log data acquired by the server

[1028] Step 3:

[1029] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy), where a speech recognition model analyzes the audio data and compiles it into text.

[1030] Input: Meeting audio data

[1031] Output: Data converted to text

[1032] Step 4:

[1033] The server analyzes the text data and extracts key points, detecting specific keywords and phrases, and capturing key information to generate a summary of the meeting content.

[1034] Input: Data converted to text

[1035] Output: Extracted key points

[1036] Step 5:

[1037] The server generates a summary of the meeting content based on the extracted key points, using a generative AI model to concisely summarize the content of the summary you want to convey.

[1038] Input: Extracted key points

[1039] Output: Summary of the meeting

[1040] Step 6:

[1041] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's speech and behavior. It analyzes characteristics such as voice pitch and word choice to extract emotional information.

[1042] Input: Meeting log data (audio and text)

[1043] Output: Recognized emotion information

[1044] Step 7:

[1045] The server reflects the recognized emotional information in the summary of the meeting, and adds emotional information to the summary, allowing participants to understand the progress of the meeting and new issues from an emotional perspective.

[1046] Input: Summary of meeting content, recognized emotion information

[1047] Output: A summary of the meeting including emotional information

[1048] Step 8:

[1049] The server updates or adds events to the project management system (e.g., JIRA) based on the summary and sentiment information, so that progress and new issues are reflected in the project management system.

[1050] Input: A summary of the meeting including emotional information

[1051] Output: Updated or added event information

[1052] Step 9:

[1053] The device notifies the user of updated event information, including progress, new tasks, and emotion information, at appropriate times.

[1054] Input: Updated or added event information

[1055] Output: Notification sent to the user

[1056] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1058] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1059] [Fourth embodiment]

[1060] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1061] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1062] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1063] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1064] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1065] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1066] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1067] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1068] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1069] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1070] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1071] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1072] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1073] The present invention provides a system for supporting efficient management of conference contents, and a specific implementation method of the system based on the roles of the server, terminal, and user will be described below.

[1074] 1. Collecting meeting log data

[1075] When a user holds a meeting, the server automatically collects the meeting log data. For example, it uses the API of the online meeting platform to obtain the content of comments made during the meeting and participant information. This allows the server to collect meeting log data that includes information such as the meeting audio, text, participant list, start time, and end time.

[1076] 2. Summary and analysis of the meeting content

[1077] The server takes the collected meeting log data and converts it into text using natural language processing technology, then extracts key points from the text data and generates a concise summary.

[1078] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing well, but Team B is facing a lack of resources."

[1079] The server then uses this summary data to track progress and identify emerging issues, making it clear whether the project is progressing as planned or where problems lie.

[1080] 3. Event updates and additions

[1081] The terminal automatically updates events on the project management system based on the progress and task information received from the server, reflecting the progress status in existing events and adding new task-based events as necessary.

[1082] For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "Lack of resources."

[1083] 4. User Notification and Response

[1084] The device notifies the user of updated events and new challenges, allowing the user to quickly understand progress and take action to address any issues that arise.

[1085] For example, a user can receive a notification, log in to a project management system to view the details, and then consider how to address the resource shortage issue and take appropriate action.

[1086] Specific examples

[1087] Suppose a user holds an online project progress meeting. After the meeting ends, the server collects the meeting log data (audio, text, participant list, etc.) through the API of the online meeting platform.

[1088] The server analyzes the collected log data, generates a summary such as "Project A is progressing smoothly, but Team B is short on resources," and based on this summary, identifies the issue of "shortage of resources for Team B."

[1089] The terminal updates the "Project A Progress" event in the project management system and adds a new event called "Lack of resources."

[1090] The terminal notifies the user of these updates, and the user can log in and check the details. The user can take measures to address resource shortages and continue the smooth progress of the project.

[1091] In this way, the system based on the present invention realizes efficient project management through the summarization and analysis of meeting contents.

[1092] The processing flow will be explained below.

[1093] Step 1:

[1094] Users use the online meeting platform to start a project status meeting, during which all statements and participant information are recorded.

[1095] Step 2:

[1096] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, including audio data, text chat, participant list, start time, end time, etc.

[1097] Step 3:

[1098] The server stores the acquired meeting log data in a database, which makes it available for later analysis.

[1099] Step 4:

[1100] The server uses natural language processing technology to convert the voice data into text, which includes everything said during the meeting.

[1101] Step 5:

[1102] The server analyzes the text data and performs processing to extract key points, for example, identifying important keywords and phrases to use as the basis for summarizing.

[1103] Step 6:

[1104] The server generates a summary of the meeting based on the extracted key points, including progress reports, issues, and new topics.

[1105] Step 7:

[1106] The server then analyzes the generated summary to identify progress and new issues, for example, "Project A is progressing smoothly, but Team B is facing a resource shortage."

[1107] Step 8:

[1108] The terminal updates events on the project management system based on the progress status and new task information received from the server, adding progress information to existing events and generating events for new tasks.

[1109] Step 9:

[1110] The device notifies the user of updated event information, including a summary of progress and new tasks, keeping the user informed quickly and accurately.

[1111] Step 10:

[1112] Users receive notifications and log into the project management system to view details, and initiate appropriate action based on the progress and issue information provided.

[1113] These are the specific processing steps of the program. This system allows users to efficiently manage the progress of projects and quickly respond to new challenges.

[1114] Example 1

[1115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1116] Conventional meeting management systems have made it difficult to effectively grasp important comments and issues raised during meetings and reflect them in subsequent project management. They also lacked a way to quickly notify users of meeting progress and new issues. This can lead to important information being overlooked during project progress, affecting the quality and progress of the final deliverables.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1118] In this invention, the server

[1119] A means of obtaining the collected meeting log data using the API of the online meeting platform;

[1120] means for converting the conference log data from voice data to text data using natural language processing technology;

[1121] means for extracting key points from the text data to generate summaries and identify progress and emerging issues;

[1122] A means for updating or adding events to a project management system based on the progress and new issues;

[1123] means for notifying a user of the updated or added event information;

[1124] This makes it possible to efficiently collect and analyze important meeting information and reflect it in the project management system. Furthermore, by promptly notifying users of progress and new issues, the project can be progressed appropriately and problems can be resolved.

[1125] "Conference log data" refers to information recorded during an online conference, such as speech content, voice data, text chat, participant list, and conference time.

[1126] An "online conference platform" is software or a service that allows multiple users to hold conferences over the Internet while sharing audio and video in real time.

[1127] "API" stands for Application Program Interface, an interface for sharing information and functions between different software systems.

[1128] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language, and is used to analyze text and voice data.

[1129] "Text data" is voice data converted into character information, and is used to facilitate analysis and display.

[1130] A "summary" is a short summary of important points extracted from the original text data.

[1131] "Progress" is information that indicates how far a project or the like has progressed relative to a plan.

[1132] An "issue" is a problem or obstacle that needs to be resolved as the project progresses.

[1133] A "project management system" is a system that centrally manages project planning, progress management, task allocation, etc., and supports efficient project management.

[1134] An "event" is an item that indicates information such as a specific progress or issue in the project management system.

[1135] "Notification" is a means by which a system notifies a user of specific information, and includes alerts and messages.

[1136] This invention is a system that supports efficient management of online meeting contents by effectively collecting and analyzing log data of online meetings and reflecting it in the related project management system. This system mainly operates based on the roles of the server, terminal, and user.

[1137] When a user starts a conference using the API of an online conference platform (e.g., an online conference application), the server automatically collects conference log data. This log data includes speech content (audio data), text chat, a participant list, the start and end times of the conference, etc. The audio data is processed to identify the speaker using speaker diarization technology.

[1138] After collecting the log data, the server stores it in storage and uses natural language processing technology to convert the voice data into text data. Specifically, it uses Google Cloud Speech-to-Text API or IBM Watson Speech to Text. The converted text data is then analyzed using Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4) to extract important points. The summary generated as a result of the analysis might include, for example, "Project A is progressing well, but Team B is facing a lack of resources." Based on this summary, the server identifies progress and new challenges.

[1139] The identified progress and issue information is sent from the server to the device. The device receives this information and automatically updates the event using the API of the project management system (e.g., project management software). Specifically, the new progress is reflected in the "Project A Progress" event, and an event based on the new issue, "Lack of Resources," is added. This process is performed through the device's API request, and the updated event information is saved in the project management system.

[1140] Updated and added event information is notified to the user via the terminal. The user can then log in to the project management system to check detailed information and take specific measures, such as allocating additional staff to resolve resource shortages.

[1141] Specific examples

[1142] Consider a case where a user holds an online project progress meeting. After the meeting ends, the server automatically collects the meeting log data (audio, text, participant list, meeting time, etc.) using the API of the online meeting platform. At this time, speaker diarization is applied to identify the speaker.

[1143] The server stores the collected log data in storage and converts the voice data into text using the Google Cloud Speech-to-Text API. The text data is then analyzed using OpenAI's GPT-4, which generates a summary that says, "Project A is progressing smoothly, but Team B's lack of resources is a problem," and identifies the "lack of resources" as a new issue.

[1144] The device receives the data sent from the server and uses the API of the project management software to update the "Project A Progress" event and add a new event called "Lack of Resources." For example, it sends an API request to add "Team B's resource shortage is an issue" to the event description.

[1145] The device notifies the user of these updates, who can then log into the project management software to view the details and take concrete action, such as allocating additional staff to address resource shortages.

[1146] Prompt Sentence Examples

[1147] Here is an example prompt for a generative AI model to explain the system in more detail:

[1148] "Please describe in detail a system that supports efficient management of meeting content. The system collects log data from online meetings and uses natural language processing techniques to summarize and identify issues. Based on this, the system updates events in a project management system and notifies users. Please also mention the specific APIs and analysis tools used."

[1149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1150] Step 1:

[1151] When a user starts an online conference, the server uses the API of the online conference platform to obtain conference log data. The obtained data includes speech content (audio data), text chat, participant list, and conference start and end times. Specifically, this data is obtained through API calls and saved in internal storage. The input is the online conference session ID, and the output is temporary saved conference log data.

[1152] Step 2:

[1153] The server converts the acquired voice data into text data using natural language processing technology. It then uses the Google Cloud Speech-to-Text API to convert the voice data into text. The input is voice data, and the output is text data. Specifically, it sends voice data to the API and receives the returned text data.

[1154] Step 3:

[1155] The server analyzes the converted text data, extracts key points, and generates a summary using the Google Cloud Natural Language API or a generative AI model (e.g., OpenAI's GPT-4). The input is the text data, and the output is a summary, identified progress, and new challenges. Specifically, the server sends the text data to the analysis engine and receives the analysis results.

[1156] Step 4:

[1157] The server organizes the identified progress and new issues along with the summary, and sends the organized data to the device. The input is the analyzed summary data, and the output is notification data to the device. Specifically, the server sends the organized data to the device's API.

[1158] Step 5:

[1159] The device automatically updates the event using the API of the project management system based on the received data. It sends an API request to update the "Project A Progress" event with the latest progress information, and generates a new event based on the new issue of "Lack of resources." The input is progress information and issue information from the server, and the output is an updated event in the project management system. Specifically, it sends an API request to update the event details.

[1160] Step 6:

[1161] The terminal notifies the user of the updated event information. The user receives the notification and logs in to the project management system to check the detailed information. The input is the event information of the project management system, and the output is the notification to the user. The specific operation is to generate a notification message and send it to the user's device.

[1162] Step 7:

[1163] Upon receiving the notification, the user accesses the project management system and takes specific measures. For example, they may take action such as adding new staff to resolve resource shortages. The input is the notified event information, and the output is the implementation of specific measures. Specifically, the user logs in to the project management system, checks the details, and records the measures.

[1164] (Application example 1)

[1165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1166] Currently, managing meeting content at manufacturing sites requires a lot of manual work, and recording and summarizing the content after meetings often takes a lot of time. This delays understanding progress and identifying new issues, making it difficult to respond quickly. Furthermore, with on-site work being hectic, there are limitations to accurately recording meeting content and quickly notifying relevant parties of the necessary information. To solve this situation, there is a need for a system that automatically collects and analyzes meeting logs and quickly notifies relevant parties of summaries and issues.

[1167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1168] In this invention, the server includes: means for acquiring collected meeting log data; means for extracting important points from the meeting log data and generating a summary; means for analyzing the summary and identifying progress and new issues; means for updating or adding events based on the progress and new issues; means for notifying the user of the updated or added event information; means installed in the smart glasses for recording the meeting log in real time; and means for automatically analyzing the meeting log to generate a summary and notifying the user of the progress and new issues. This improves the efficiency of meeting content management at factory manufacturing sites, enabling quick progress understanding and issue identification.

[1169] "Collected conference log data" refers to data that includes information such as audio recorded during a conference, text, a participant list, start time, and end time.

[1170] "Important points" refer to statements, issues, decisions, etc. that are considered particularly important within the content of the meeting.

[1171] A "summary" is a document or textual information that briefly summarizes what was discussed at a meeting.

[1172] "Progress" refers to the current state or stage of a project or task.

[1173] "New issues" refer to new problems or issues that need to be addressed that were identified during the meeting.

[1174] "Updating or adding an event" means modifying or creating new tasks or schedules in an existing project management system based on the latest information.

[1175] "Notifying users" means informing users of the generated update information and information about new issues via email, a notification system, or the like.

[1176] "Smart glasses" are a type of wearable device that refers to a glasses-type display device that has the ability to display and record information in real time.

[1177] "Real-time recording" means collecting audio and video simultaneously while the meeting is in progress and saving the data.

[1178] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[1179] The present invention relates to a system for efficiently managing meetings at a manufacturing site in a factory. The system is configured based on the roles of smart glasses, a server, a terminal, and a user.

[1180] Meeting log data collection

[1181] First, when a user holds a meeting, the smart glasses record the meeting log data in real time. The smart glasses are equipped with speech recognition and text conversion functions, and automatically collect speech data during the meeting and convert it into text data. This data is then sent to a server.

[1182] Meeting summary and analysis

[1183] The server receives the meeting log data sent from the smart glasses and converts it into text using natural language processing technology. It then extracts key points from the collected text data and generates a concise summary. The server uses the SpaCy model as its natural language processing technology. For example, it generates a summary such as, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[1184] Event updates and additions

[1185] The terminal automatically updates events on the project management system based on the progress and issue information received from the server. It reflects the progress status in existing events and adds events based on new issues as necessary. For example, when progress information is updated, the terminal adds the latest progress report to an event such as "Progress of Project A" and generates an event to address the newly identified issue of "lack of resources."

[1186] User notification and response

[1187] The device notifies the user of updated events and new tasks, allowing the user to quickly understand the progress and take action to address any issues that arise. Specifically, email notifications are sent using Python's smtplib. For example, notifications are sent to email addresses such as "user1@example.com" and "user2@example.com."

[1188] Specific examples

[1189] For example, a regular meeting is held in the manufacturing department of a factory, and after the meeting, smart glasses collect the meeting log and generate a summary such as "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials." Based on this, a new issue is added to the project management system and the user is notified about it by email.

[1190] Prompt Sentence Examples

[1191] Generate a meeting log stating, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[1192] This approach streamlines meeting content management on the factory floor, allowing for quick progress updates and the identification of emerging issues, while also enabling more advanced analysis and automation using generative AI models and prompts.

[1193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1194] Step 1:

[1195] Collection and transmission of meeting log data

[1196] A user wears smart glasses and holds a conference. The glasses collect audio data during the conference in real time and convert it into text data. This text data includes the content of the speech, the speaker, a timestamp, etc. The smart glasses then send this data to a server.

[1197] Input: Audio data during the meeting

[1198] Output: Meeting log data in text format

[1199] Step 2:

[1200] Receiving and storing meeting log data

[1201] The server receives the meeting log data sent from the smart glasses and stores it in a database for later analysis.

[1202] Input: Text data sent from smart glasses

[1203] Output: Meeting log data stored in a database

[1204] Step 3:

[1205] Meeting summary generation

[1206] The server analyzes the saved meeting log data using natural language processing technology. It uses a generative AI model (e.g., SpaCy) to extract key points and generate a summary. For example, a summary might be generated that reads, "Production line 1 is progressing smoothly, but production line 2 is running low on raw materials."

[1207] Input: Meeting log data stored in a database

[1208] Output: Summary of meeting content

[1209] Step 4:

[1210] Identifying progress and new challenges

[1211] The server identifies progress and new issues from the generated summary, and uses natural language processing techniques to extract problem and progress information from keywords and phrases in the summary.

[1212] Input: Summary of meeting content

[1213] Output: Identified progress and challenges

[1214] Step 5:

[1215] Event updates and additions

[1216] The terminal automatically updates events in the project management system based on the identified progress and issue information. It reflects the progress in existing events and adds new events based on issues as needed. For example, it adds a new resource arrangement event based on the fact that "production line 2 is short of raw materials."

[1217] Input: Identified progress and challenges

[1218] Output: Updated or added event information

[1219] Step 6:

[1220] User Notification

[1221] The device notifies the user of updated events and new challenges via email or push notifications on the smartphone app, allowing the user to quickly understand progress and address any issues.

[1222] Input: Updated or added event information

[1223] Output: User notification

[1224] These steps allow for efficient management of meeting content and enable rapid response on-site.

[1225] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1226] This invention combines a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support efficient management of meeting content with an emotion engine that recognizes user emotions. Below, we will explain specific implementation methods of this invention based on the roles of the server, terminal, and user.

[1227] 1. Collecting meeting log data

[1228] Users use the online meeting platform to hold project progress meetings, and all statements, text chats, and participant information are recorded by the platform.

[1229] 2. Acquiring meeting log data

[1230] After the meeting ends, the server retrieves the meeting log data through the online meeting platform's API. The retrieved data includes audio data, text data, participant list, start time, and end time. The retrieved meeting log data is then stored in a database.

[1231] 3. Summary and analysis of the meeting content

[1232] The server uses natural language processing technology to convert the voice data into text, analyzes the converted text data, extracts important points, and generates a summary of the meeting content based on the extracted points.

[1233] For example, the server analyzes the text content of a meeting and creates a summary such as, "Project A is progressing smoothly, but Team B is experiencing a resource shortage."

[1234] 4. Emotion Recognition by Emotion Engine

[1235] During the analysis of the meeting log data, the server uses an emotion engine to recognize emotions from the user's remarks and attitudes. For example, it can determine whether the user is feeling stressed from the pitch of their voice and the choice of words.

[1236] The perceived emotional information is reflected in the meeting summary, and additional information is added, for example, in the form of "Team B's leader expresses strong concerns about the lack of resources."

[1237] 5. Event updates and additions

[1238] The terminal automatically updates and adds events to the project management system based on the progress, new issues, and emotion information sent from the server. For example, the terminal adds the latest progress report to the "Project A Progress" event and generates an event for the new issue of "Lack of resources."

[1239] 6. User Notification and Response

[1240] The device notifies the user of updated event information, including progress, new tasks, and user emotional information, at the most appropriate time.

[1241] For example, it is considered that by notifying the user when he or she is relaxed, the user can calmly deal with the problem.

[1242] Specific examples

[1243] Suppose a user holds an online project progress meeting. After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform.

[1244] The server analyzes the log data and converts it into text using natural language processing technology. It extracts key points and generates a summary such as, "Project A is progressing smoothly, but Team B is facing a lack of resources."

[1245] Furthermore, the server uses an emotion engine to extract emotional information from the log data and add information such as "Team B members are feeling very stressed about the lack of resources."

[1246] The device updates the events on the project management system based on the received information, generates a new event for the issue "resource shortage," and sends a notification to the user while taking into account emotional information.

[1247] Users receive notifications, log in to their project management systems to review the details, consider how to address the resource shortage, and initiate appropriate actions.

[1248] The present invention enables efficient management of meeting content including emotional information, and enables prompt and accurate responses to problems.

[1249] The processing flow will be explained below.

[1250] Step 1:

[1251] Users use an online meeting platform to initiate a project status meeting, during which all statements, text chats, and participant information are recorded.

[1252] Step 2:

[1253] After the meeting ends, the server retrieves the meeting log data through the API of the online meeting platform, which includes audio data, text data, a list of participants, and the start and end times.

[1254] Step 3:

[1255] The server stores the acquired log data in a database so that it can be used for later analysis.

[1256] Step 4:

[1257] The server uses natural language processing technology to convert the voice data into text, which includes all the content spoken during the meeting.

[1258] Step 5:

[1259] The server analyzes the converted text data, extracts important points and keywords from the meeting, and generates a summary based on the extracted points.

[1260] Step 6:

[1261] The server uses the summary to identify progress and emerging issues, such as "Project A is progressing well, but Team B is facing a lack of resources."

[1262] Step 7:

[1263] The server uses an emotion engine to recognize users' emotions from meeting log data, for example, determining whether they are stressed based on their voice pitch and word choice.

[1264] Step 8:

[1265] The server adds the recognized emotional information to the meeting summary, which includes emotional information such as "Team B members are highly stressed about the lack of resources."

[1266] Step 9:

[1267] The terminal automatically updates and adds events to the project management system based on the progress status, new tasks, and emotion information received from the server. Existing events are updated with the latest progress information, and new events are generated for new tasks.

[1268] Step 10:

[1269] The device notifies the user of updated event information, including progress, new tasks, and emotional information, and the content and timing of notifications are tailored to the user's emotional state.

[1270] Step 11:

[1271] Users receive notifications, log into their project management system to review details, and initiate appropriate action based on the progress, issue information, and sentiment information provided.

[1272] These are the specific processing steps based on the present invention, which combines an emotion engine. This system allows users to efficiently manage project progress and quickly respond to new challenges. In addition, taking emotion information into account promotes optimal behavior for problem solving.

[1273] Example 2

[1274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] Conventional meeting management systems have difficulty efficiently grasping the progress and issues of meetings, and more accurate decision-making is required, especially by including participants' emotional information. Furthermore, since summarizing meeting content and extracting key points takes time and effort, automation is necessary. Furthermore, project management updates and notifications based on emotional information are also an important issue.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1277] In this invention, the server includes means for acquiring collected meeting log data, means for converting the meeting log data into voice data and text data, means for extracting important points from the voice data using natural language processing technology and generating a summary, means for extracting user emotional information from the text data and voice data using emotion analysis technology and incorporating the information into the summary, means for analyzing the summary to identify progress and new issues, means for updating or adding events to the project management system based on the progress and new issues, and means for notifying the user of the updated or added event information and emotional information. This enables efficient management of meeting content, accelerated decision-making including emotional information, and automatic updating and notification of the project management system.

[1278] "Conference log data" refers to data recorded during a conference, including statements made during the conference, text chat, participant information, and the start and end times of the conference.

[1279] "Audio data" refers to data recorded in audio file format of what was said during a conference.

[1280] "Text data" refers to text-format data converted from voice data, or text chat data recorded during a conference.

[1281] "Natural language processing technology" is a technology for analyzing language data and understanding its meaning, and in the present invention, it is a technology used as a means for converting voice data into text and extracting important points.

[1282] "Emotion analysis technology" is a technology that recognizes and extracts the emotional state of a speaker from text data and audio data.

[1283] The "summary" is a concise summary of the important points of the meeting content extracted using natural language processing technology.

[1284] "Progress" refers to the current state or progress of a project or task.

[1285] "Issues" are problems that arise during the project or areas that require improvement.

[1286] An "event" is an entry in a project management system that records and tracks actions on progress or new issues.

[1287] A "project management system" is a software system for planning, executing, and monitoring tasks or projects.

[1288] "Communication means" refers to the technology and methods by which servers and terminals exchange data with online meeting platforms and project management systems.

[1289] "Notification" is the act of notifying the user of updated event information or important messages.

[1290] The present invention provides a system for efficiently managing meeting log data collected using an online meeting platform and generating a detailed summary including user emotional information. In the following embodiments, specific implementation methods will be described based on the roles of the server, terminal, and user.

[1291] 1. Overview of the entire system

[1292] This system supports large-scale project management and efficient meeting management for remote teams. The system acquires meeting log data from an online meeting platform and analyzes it using natural language processing and sentiment analysis technologies. As a result, it is possible to grasp a summary of the meeting content, progress, new issues, and user sentiment. This automatically updates events in the project management system and sends notifications to users at optimal times.

[1293] 2. Hardware and Software Configuration

[1294] server

[1295] The server is a general-purpose computer with a powerful processor and sufficient memory, running the following software:

[1296] Operating System: Linux

[1297] Database: MySQL

[1298] Natural language processing technology: Google Cloud Speech-to-Text API, spaCy

[1299] Emotion analysis technology: Amazon Comprehend

[1300] Terminal

[1301] The terminal is a personal computer or mobile device operated by a user, running the following software:

[1302] Operating Systems: Windows, macOS, Android, iOS

[1303] Project management systems: JIRA, Asana

[1304] User

[1305] Users are project managers and meeting participants who participate in meetings through an online meeting platform. In particular, the following environment is required:

[1306] Online meeting platforms: Zoom, Microsoft Teams

[1307] 3. Specific examples of implementation

[1308] A user holds a project status meeting using Zoom. After the meeting ends, the server retrieves the meeting log data through the Zoom API. The server converts the acquired audio data into text using the Google Cloud Speech-to-Text API, analyzes the converted text data using natural language processing technology (spaCy), and extracts key points. The server then extracts sentiment information using Amazon Comprehend and combines the summary with the sentiment information.

[1309] Specific prompt examples:

[1310] "Generate a summary of a project status meeting. Analyze the following log data and include key points and sentiment information."

[1311] 4. Operational Flow

[1312] A user holds a meeting, and the meeting log data is recorded by the online meeting platform. After the meeting ends, the server retrieves the recorded data and performs natural language processing and sentiment analysis. A summary is generated based on the analysis results, and the event is updated and added to the project management system. The terminal notifies the user of the update, and the user can access the project management system to check the details.

[1313] The present invention enables efficient management of meeting content and quick and accurate responses that take into account user feelings, thereby increasing the success rate of projects.

[1314] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1315] Step 1: Hosting a meeting and collecting log data

[1316] A user uses an online conference platform to hold a project progress meeting. When the user starts the meeting, the online conference platform records the remarks made during the meeting, text chat, and participant information in real time. The input is the user's voice and text remarks, and the output is the meeting log data. This log data is stored on the server of the online conference platform.

[1317] Step 2: Retrieving meeting log data

[1318] After a meeting ends, the server retrieves the meeting log data through the API of the online meeting platform. Specifically, the server sends a request to the API endpoint to retrieve the meeting log data, including audio data, text data, a list of participants, and the start and end times. The input is the request to the API, and the output is the retrieved dataset. This dataset is then stored in a database.

[1319] Step 3: Transcribe the meeting content

[1320] The server converts the audio data into text using the Google Cloud Speech-to-Text API. The server sends the acquired audio data to the API and receives the corresponding text data. The input is audio data and the output is text data. This text data is saved as a temporary file and used in the next step.

[1321] Step 4: Analyze text data and generate summaries

[1322] The server uses an NLP (Natural Language Processing) library (e.g. spaCy) to analyze the converted text data and extract key points. The server analyzes the text data and extracts the main points of the meeting. The input is the text data, and the output is a list of key points. A summary is generated based on these key points. The server aggregates the extracted points and generates a summary of the meeting content. The output is the generated summary.

[1323] Step 5: Performing Emotion Recognition

[1324] The server sends text and audio data to the emotion engine Amazon Comprehend to extract the user's emotional information. The server passes the acquired data to the emotion engine and receives the analysis results. The input is text and audio data, and the output is emotional information. This emotional information is added to the summary. For example, information such as "The leader of Team B expresses strong concerns about the lack of resources" is added.

[1325] Step 6: Update your project management system

[1326] The device updates the project management system based on the progress, new tasks, and sentiment information sent from the server. The device receives data from the server and updates and adds events to the project management system through API. The input is the data received from the server, and the output is the updated event in the project management system. For example, a new report is added to "Project A Progress."

[1327] Step 7: Notify users

[1328] The device notifies the user of the updated event information. Based on the updated information, the device sends a notification to the user in an appropriate format (e.g., push notification, email, dashboard display). The input is the updated event information, and the output is a notification to the user. The user receives the notification and logs in to the project management system to check the detailed information. For example, specific instructions such as "Please start considering countermeasures for the resource shortage issue" are displayed.

[1329] The system enables efficient management of meeting content, generation of detailed summaries including emotional information, and automatic updates and notifications for project management systems.

[1330] (Application example 2)

[1331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] In production management and maintenance meetings at factories, it is necessary to efficiently grasp progress and identify new issues. However, typical meeting recording systems are limited to summarizing meetings and extracting progress, and are unable to consider the emotions and stress levels of meeting participants. This poses a challenge, making it difficult to provide more detailed and practical problem-solving based on emotional information. In particular, in factories, where resource shortages and emergency responses are often required, a rapid response that takes into account the emotions of participants is required.

[1333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1334] In this invention, the server includes means for acquiring collected meeting log data, means for extracting important points from the meeting log data and generating a summary, means for recognizing emotions from user comments and attitudes, means for reflecting the emotional information in the summary, means for updating or adding events based on the progress and new tasks, and means for notifying the user of the updated or added event information. This enables detailed analysis that reflects emotional information along with a summary of the meeting content, enabling quick and practical problem-solving in factory production management and maintenance.

[1335] "Collected meeting log data" refers to data recorded during a meeting, such as audio data, text chat, participant information, and the start and end times of the meeting.

[1336] "Extracting key points" refers to extracting specific keywords and phrases from the meeting log data and extracting important information to create a summary of the meeting content.

[1337] "Summary generation" refers to creating a short and concise summary of the entire meeting based on the extracted important points.

[1338] "Identifying progress" refers to understanding the progress of projects and tasks discussed in meetings and clarifying their status and degree of achievement.

[1339] "Identifying new issues" refers to identifying new problems or unresolved issues that arise during the meeting.

[1340] "Updating or adding events" refers to updating action items or tasks in your project management system or adding new events based on the progress of the meeting or new issues.

[1341] "Recognizing emotions from users' comments and attitudes" refers to detecting participants' emotional states by analyzing the audio and text of meeting log data.

[1342] "Reflecting emotional information in summaries" means including recognized emotional information in the summary content to provide detailed meeting content that takes into account changes in emotions and signs of stress.

[1343] "Notifying the user" refers to reporting updated or added event information, progress, new challenges, and emotional information to the user at an appropriate time.

[1344] This invention is a system that collects and analyzes meeting log data, grasps progress, and identifies new issues to support the efficient management of production meetings and maintenance meetings in factories. It also incorporates an emotion engine that recognizes the user's emotions. Below, we will explain how to implement this invention based on the roles of the server, terminal, and user.

[1345] 1. Collecting meeting log data

[1346] Users hold production meetings and maintenance meetings in the factory. All audio, text chat, and participant information from the meetings are recorded by the factory robot, which is equipped with a microphone, camera, and text chat collection tool.

[1347] 2. Acquiring meeting log data

[1348] After the meeting ends, the server retrieves the meeting log data collected from the factory robot. The retrieved data includes voice data, text data, a list of participants, and the start and end times. This data is then stored in a database.

[1349] 3. Summary and analysis of the meeting content

[1350] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy). The converted text data is then analyzed to extract key points. This process involves extracting specific keywords and important phrases from the verbatim audio data and generating a summary based on that. For example, the summary generated might say, "Project A is progressing well, but Team B is short on resources."

[1351] 4. Emotion Recognition by Emotion Engine

[1352] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's remarks and attitudes. It determines whether the user is feeling stressed based on the pitch of the voice data and the context of the text data. The recognized emotion information is reflected in the meeting summary, adding information such as, "The leader of Team B is feeling very stressed about the lack of resources."

[1353] 5. Event updates and additions

[1354] The server automatically updates and adds events to the project management system (e.g., JIRA) based on the analyzed progress, new issues, and sentiment information. For example, it adds the latest progress report to the "Project A Progress" event and generates an event for a new issue called "Lack of Resources."

[1355] 6. User Notification and Response

[1356] The device notifies the user of updated event information. Notifications include progress, new tasks, and emotional information. Notifications are sent at the most appropriate time, taking into account the user's state. For example, by notifying the user when they are relaxed, the device allows them to respond calmly to problems.

[1357] Specific examples

[1358] A regular meeting is held at a factory, and a factory robot records the meeting. After the meeting ends, the server obtains the data collected from the robot and converts it into text using natural language processing technology. It extracts important points and generates a summary such as, "Project A is progressing smoothly, but Team B's lack of resources is an issue." The server then uses an emotion engine to extract emotional information from the log data and adds information such as, "Team B members are feeling very stressed about the lack of resources." The terminal uses this information to update an event on the project management system (JIRA) and generates a new event for the issue, "Lack of Resources." It also sends a notification to the user, taking the emotional information into consideration. The user receives the notification, logs in to the project management system to check the details, considers countermeasures for the resource shortage, and initiates appropriate action.

[1359] Prompt Sentence Examples

[1360] Convert speech data to text, analyze the text for key points and sentiment, and use the results to update events in your project management system and add new issues.

[1361] As follows:

[1362] 1. Analyze the audio data and convert it into text.

[1363] 2. The converted text is analyzed using IBM Watson's sentiment analysis API.

[1364] 3. Based on the analysis results, add a new event to the project management system (JIRA) and send a notification.

[1365] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1366] Step 1:

[1367] Users hold production meetings and maintenance conferences in the factory. The robot uses a microphone, camera, and text chat collection tool to collect voice data, text chat data, participant information, and the start and end times of the conference.

[1368] Input: All meeting audio, text chat, and participant information

[1369] Output: Collected meeting log data

[1370] Step 2:

[1371] After the meeting ends, the server acquires the meeting log data collected from the robot.

[1372] Input: Meeting log data stored in the robot

[1373] Output: Meeting log data acquired by the server

[1374] Step 3:

[1375] The server converts the audio data into text using natural language processing techniques (e.g., NLTK or SpaCy), where a speech recognition model analyzes the audio data and compiles it into text.

[1376] Input: Meeting audio data

[1377] Output: Data converted to text

[1378] Step 4:

[1379] The server analyzes the text data and extracts key points, detecting specific keywords and phrases, and capturing key information to generate a summary of the meeting content.

[1380] Input: Data converted to text

[1381] Output: Extracted key points

[1382] Step 5:

[1383] The server generates a summary of the meeting content based on the extracted key points, using a generative AI model to concisely summarize the content of the summary you want to convey.

[1384] Input: Extracted key points

[1385] Output: Summary of the meeting

[1386] Step 6:

[1387] The server uses an emotion engine (e.g., IBM Watson's emotion analysis API) to recognize emotions from the user's speech and behavior. It analyzes characteristics such as voice pitch and word choice to extract emotional information.

[1388] Input: Meeting log data (audio and text)

[1389] Output: Recognized emotion information

[1390] Step 7:

[1391] The server reflects the recognized emotional information in the summary of the meeting, and adds emotional information to the summary, allowing participants to understand the progress of the meeting and new issues from an emotional perspective.

[1392] Input: Summary of meeting content, recognized emotion information

[1393] Output: A summary of the meeting including emotional information

[1394] Step 8:

[1395] The server updates or adds events to the project management system (e.g., JIRA) based on the summary and sentiment information, so that progress and new issues are reflected in the project management system.

[1396] Input: A summary of the meeting including emotional information

[1397] Output: Updated or added event information

[1398] Step 9:

[1399] The device notifies the user of updated event information, including progress, new tasks, and emotion information, at appropriate times.

[1400] Input: Updated or added event information

[1401] Output: Notification sent to the user

[1402] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1404] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1405] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1406] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1407] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1408] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1409] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1410] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1411] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1412] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1413] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1414] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1415] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1416] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1417] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1418] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1419] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1420] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1421] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1422] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1423] The following is further disclosed regarding the above embodiment.

[1424] (Claim 1)

[1425] a means for obtaining the collected meeting log data;

[1426] means for extracting important points from the meeting log data and generating a summary;

[1427] means for analyzing said summaries to identify progress and emerging issues;

[1428] means for updating or adding events based on said progress and new challenges;

[1429] means for notifying a user of the updated or added event information;

[1430] A system including:

[1431] (Claim 2)

[1432] 10. The system of claim 1, further comprising a communication means for acquiring the conference log data.

[1433] (Claim 3)

[1434] 10. The system of claim 1, further comprising means for using natural language processing techniques to analyze the conference log data.

[1435] "Example 1"

[1436] (Claim 1)

[1437] A means of obtaining the collected meeting log data using the API of the online meeting platform;

[1438] means for converting the conference log data from voice data to text data using natural language processing technology;

[1439] means for extracting key points from the text data to generate summaries and identify progress and emerging issues;

[1440] A means for updating or adding events to a project management system based on the progress and new issues;

[1441] means for notifying a user of the updated or added event information;

[1442] A system including:

[1443] (Claim 2)

[1444] 10. The system of claim 1, further comprising a communication means for acquiring the conference log data.

[1445] (Claim 3)

[1446] 10. The system of claim 1, further comprising means for using natural language processing techniques to analyze the conference log data.

[1447] "Application Example 1"

[1448] (Claim 1)

[1449] a means for obtaining the collected meeting log data;

[1450] means for extracting important points from the meeting log data and generating a summary;

[1451] means for analyzing said summaries to identify progress and emerging issues;

[1452] means for updating or adding events based on said progress and new challenges;

[1453] means for notifying a user of the updated or added event information;

[1454] A means for recording meeting logs in real time, installed on smart glasses;

[1455] means for automatically analyzing the meeting log to generate a summary and notify the user of progress and new issues;

[1456] A system including:

[1457] (Claim 2)

[1458] 10. The system of claim 1, further comprising a communication means for acquiring the conference log data.

[1459] (Claim 3)

[1460] 10. The system of claim 1, further comprising means for using natural language processing techniques to analyze the conference log data.

[1461] "Example 2: Combining Emotion Engines"

[1462] (Claim 1)

[1463] a means for obtaining the collected meeting log data;

[1464] means for converting the conference log data into voice data and text data;

[1465] means for extracting key points from the speech data using natural language processing techniques and generating a summary;

[1466] means for extracting user emotion information from the text data and the audio data using emotion analysis technology and incorporating the information into the summary;

[1467] means for analyzing said summaries to identify progress and emerging issues;

[1468] means for updating or adding events to a project management system based on the progress and new issues;

[1469] means for notifying a user of the updated or added event information and emotion information;

[1470] A system including:

[1471] (Claim 2)

[1472] 10. The system of claim 1, further comprising a communication means for acquiring the conference log data.

[1473] (Claim 3)

[1474] 10. The system of claim 1, further comprising means for using natural language processing and sentiment analysis techniques to analyze the meeting log data.

[1475] "Application example 2 when combining emotion engines"

[1476] (Claim 1)

[1477] a means for obtaining the collected meeting log data;

[1478] means for extracting important points from the meeting log data and generating a summary;

[1479] means for analyzing said summaries to identify progress and emerging issues;

[1480] means for updating or adding events based on said progress and new challenges;

[1481] A means for recognizing emotions from user statements and attitudes;

[1482] means for reflecting the emotion information in the summary;

[1483] means for notifying a user of the updated or added event information;

[1484] A system including:

[1485] (Claim 2)

[1486] 10. The system of claim 1, further comprising a communication means for acquiring the conference log data.

[1487] (Claim 3)

[1488] 10. The system of claim 1, further comprising means for using natural language processing techniques to analyze the conference log data. [Explanation of symbols]

[1489] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining the collected meeting log data; means for extracting important points from the meeting log data and generating a summary; means for analyzing said summaries to identify progress and emerging issues; means for updating or adding events based on said progress and new challenges; means for notifying a user of the updated or added event information; A system including:

2. 2. The system of claim 1, further comprising a communication means for obtaining said conference log data.

3. The system of claim 1 further comprising means for using natural language processing techniques to analyze the conference log data.

Images