System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting meeting records; means for analyzing the collected meeting records with a natural language processing model and generating a summary; means for generating a question-answer pair from the meeting records and the generated summary and storing the pair in a database; means for predicting a topic to be discussed next based on past meeting records; means for displaying the generated summary to a user terminal; means for retrieving an answer from the database based on a question from the user terminal and displaying the answer; and means for displaying the topic to be discussed next on the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern business activities, meetings are important for decision-making and information sharing. However, differences in the timing and background of each participant can lead to differences in understanding, which can hinder meetings from proceeding smoothly. In particular, when participants arrive late or are inadequately prepared, re-explanations or additional explanations are required, wasting time and effort. As a result, meeting productivity declines and important decision-making is often delayed. The present invention aims to provide technology that solves these problems and improves meeting efficiency and productivity. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means: A means for automatically collecting meeting records, and a means for analyzing the collected meeting records using a natural language processing model and generating summaries. Furthermore, a means for generating question and answer pairs from the meeting records and the generated summaries and saving them in a database. A means for predicting the next topic to be discussed based on past meeting records, and a means for displaying the generated summary on a user terminal, and a means for searching for and displaying an answer from a database based on a question from the user terminal. Furthermore, by providing a means for displaying the next topic to be discussed on a user terminal, a system is provided that supports the progress of a meeting, eliminates differences in understanding among participants, and improves meeting productivity.

[0006] "Meeting records" refers to all information generated during a meeting (audio, video, minutes, etc.), including the content of discussions and conclusions.

[0007] "Means of collection" refers to methods and technologies for automatically obtaining meeting records from a conferencing system.

[0008] "Natural language processing model" refers to the techniques and algorithms used by computers to understand, parse, and generate human language.

[0009] A "summary" refers to information that extracts important points and conclusions from a meeting record and presents them in a shortened form.

[0010] A "question and answer pair" is a combination of a possible question and its answer generated from the conference record, and means information used for responses in the search system.

[0011] "Database" refers to an organized collection of information, particularly a system in which question and answer pairs and other related information are stored.

[0012] "Predictive tools" refers to techniques and methods that use past data and patterns to predict future events or topics.

[0013] "User terminal" refers to a device such as a computer, smartphone, or tablet that is used to operate the system.

[0014] "Display means" refers to methods and techniques for visually presenting information to a user terminal, including a screen or display. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system aimed at improving meeting productivity, and its main functions include collecting meeting records, generating summaries using natural language processing, generating question and answer pairs, building and searching a database, predicting the next discussion topic, and providing this information to user terminals.

[0037] Server-side processing

[0038] 1. Collection of meeting records

[0039] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[0040] Examples:

[0041] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[0042] 2. Meeting Record Analysis and Summary Generation

[0043] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[0044] Examples:

[0045] The audio file is converted into text, key discussion points are extracted, and a summary is generated with "budget cuts" and "marketing strategies for new products" as the main topics.

[0046] 3. Building a QA database

[0047] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[0048] Examples:

[0049] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[0050] 4. Predicting the next discussion topic

[0051] The server uses past meeting recordings to train a machine learning model to predict topics that are likely to be discussed in the next meeting, allowing users to efficiently prepare for their next meeting.

[0052] Examples:

[0053] By studying past discussion patterns, it predicts that "increasing the online advertising budget" is likely to be discussed at the next meeting.

[0054] Terminal side processing

[0055] 1. View the meeting summary

[0056] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[0057] Examples:

[0058] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[0059] 2. Q&A search function

[0060] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[0061] Examples:

[0062] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[0063] 3. Display next discussion topic

[0064] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[0065] Examples:

[0066] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[0067] User processing

[0068] 1. Check the summary

[0069] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[0070] Examples:

[0071] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[0072] 2. Using Q&A Search

[0073] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[0074] Examples:

[0075] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[0076] 3. Confirm the next discussion topic

[0077] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[0078] Examples:

[0079] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[0080] The present invention allows each participant to quickly obtain necessary information in real time, greatly improving the productivity of the meeting.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing system (e.g., Zoom or MICROSOFT (registered trademark) TEAMS (registered trademark)). Specifically, it uses an API to set up the server to retrieve data as soon as the meeting ends.

[0084] Step 2:

[0085] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google® Cloud Speech-to-Text API, which automatically converts the voice recording into text.

[0086] Step 3:

[0087] The server applies natural language processing (NLP) models to analyze the converted text data, which involves extracting key discussion points and keywords within the text. For example, an IBM Watson® NLP model is used to identify key discussion points.

[0088] Step 4:

[0089] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main discussion points and conclusions. For example, "budget cuts and new product marketing strategies" is summarized as the main discussion points.

[0090] Step 5:

[0091] The server generates question and answer pairs (QA pairs) based on the generated summary and meeting transcripts. This process involves using natural language generation (NLG) technology to create appropriate answers to anticipated questions.

[0092] Step 6:

[0093] The server stores the generated Q&A pairs in a database, allowing it to provide quick answers when users ask questions later.

[0094] Step 7:

[0095] The server uses past meeting recordings and generated summaries to predict topics that may be discussed in the next meeting, which involves using machine learning models to learn past patterns and infer future topics.

[0096] Step 8:

[0097] When a user receives a notification of the next meeting, the terminal accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[0098] Step 9:

[0099] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays the answers to the user.

[0100] Step 10:

[0101] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[0102] Step 11:

[0103] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] In today's business world, improving the productivity of meetings is extremely important. However, manually organizing huge amounts of meeting records, creating summaries, and predicting the next discussion topic is extremely time-consuming and labor-intensive. Another problem is the lack of a way to efficiently review meeting content and quickly retrieve information from QA databases. Therefore, a new system is needed to solve these problems and improve meeting efficiency.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for acquiring meeting records using an API, means for converting voice data to text using a speech recognition API, means for extracting main points of the discussion and generating summaries using a natural language processing model, and means for predicting the next topic to be discussed using a machine learning model. This enables automatic collection of meeting records, generation of summaries, generation and search of Q&A pairs, and prediction of the next topic to be discussed.

[0109] "Meeting records" are data that record what was discussed, what was said, and what decisions were made at a meeting.

[0110] "Natural language processing model" refers to an algorithm or machine learning model used to understand, generate, or parse human language.

[0111] "API" stands for Application Programming Interface, a set of rules and protocols that allow software to communicate with one another.

[0112] A "speech recognition API" is an application programming interface that provides functionality for analyzing voice data and converting it into text.

[0113] "Summary generation" is the process of extracting important information from long text data and summarizing it concisely.

[0114] A "question and answer pair (QA pair)" is a combination of data that associates a suitable answer to a specific question.

[0115] A "database" is a system for systematically storing and managing large amounts of data, and for efficiently accessing, updating, and searching them.

[0116] A "machine learning model" is a collection of algorithms that learn patterns and rules based on large amounts of data and make predictions and classifications.

[0117] "User terminal" refers to a device through which a user accesses information and services, including a PC, smartphone, tablet, etc.

[0118] A "summary" is a concise expression of the main points extracted from detailed information.

[0119] "Searching" is the process of locating specific information in a database or other data source.

[0120] "Topic prediction" is the process of using past data to predict topics that are likely to be covered in future meetings or discussions.

[0121] The present invention is a system for improving the productivity of meetings, and specifically includes the following functions: collecting meeting records, generating summaries using natural language processing, generating question-and-answer pairs, building and searching a database, predicting the next topic of discussion, and providing this information to a user terminal. Below, we will explain the specific procedures for implementing the present invention, as well as the hardware and software used.

[0122] Server Processing

[0123] The server first collects meeting records from the company's conferencing system (e.g., video conferencing, audio conferencing) using an API. For example, the server uses Zoom's API to download the minutes and audio data after the meeting ends. Next, the downloaded audio data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). This text data is then analyzed using a natural language processing model (e.g., the BERT model) to extract the main points of the discussion and generate a summary.

[0124] Based on the generated summaries and meeting records, the server creates question-and-answer pairs (QA pairs). It uses a question generation model to extract questions from text data and uses information retrieval algorithms to obtain appropriate answers. The generated QA pairs are stored in a database.

[0125] Furthermore, the server uses past meeting records to train a machine learning model (e.g., LSTM) to predict the next discussion topic. This model learns past discussion patterns and predicts the topics that are likely to be discussed in the next meeting. The prediction results are also stored in the database.

[0126] Terminal handling

[0127] When a user accesses the meeting tool from their device, the server sends the latest meeting summary to the device and displays it. Specifically, the device sends an API request to the server to retrieve the summary data, and then displays this data to the user.

[0128] When a user inputs a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. The device sends a search query and the best answer is returned and displayed.

[0129] Additionally, the next meeting preparation screen displays the next discussion topic retrieved from the server by the predictive model, allowing users to efficiently prepare for the next meeting.

[0130] User Action

[0131] Users can check the summary before a meeting to quickly understand the content of past meetings. For example, even if they join a meeting late, they can quickly understand the main points discussed by checking the meeting summary.

[0132] During meetings or in preparation, users can use the Q&A search function to instantly obtain the information they need. For example, if they ask, "When is the new product released?", the answer displayed will be "It is scheduled for March 2024."

[0133] Next discussion topics are predicted using machine learning models and displayed to users, allowing them to prepare in advance for topics that are likely to be discussed in the next meeting, such as preparing materials on "online advertising budgets."

[0134] Through these procedures, the present invention can increase the efficiency of meetings and allow users to quickly obtain the information they need.

[0135] Generative AI model and example prompts

[0136] For example, the prompt to the user to get a summary for a particular conference is:

[0137] "View the summary of the weekly meeting for September 30th."

[0138] In this way, the present invention combines various technologies (APIs, speech recognition APIs, natural language processing models, and machine learning models) to automate meeting recording, summary generation, Q&A pair creation, and topic prediction, providing users with an efficient means of obtaining information.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] Collection of meeting records

[0142] The server calls the API to retrieve the meeting records. Specifically, the server uses the API key to send a request to the endpoint of the conference system (e.g., a video conference system). The conference system's API returns the text minutes and audio data based on the conference ID. The server stores the returned data in local storage.

[0143] Input: Meeting ID, API key

[0144] Output: Text transcript, audio data

[0145] Step 2:

[0146] Analysis of meeting recordings and transcription of audio data

[0147] The server converts the acquired voice data into text data using a voice recognition API (e.g., Google Cloud Speech-to-Text). Specifically, it sends the voice data to the voice recognition API and saves the returned text data in local storage.

[0148] Input: Audio data

[0149] Output: Text data

[0150] Step 3:

[0151] Meeting Record Analysis and Summary Generation

[0152] The server uses a natural language processing model (e.g., the BERT model) to analyze the text data, extract key arguments and conclusions, and generate a summary. Specifically, the text data is input into the natural language processing model, which extracts the main points of the argument and outputs them as a summary. The server then stores the generated summary in a database.

[0153] Input: Text data

[0154] Output: Summary

[0155] Step 4:

[0156] Building a QA database

[0157] The server generates question and answer pairs (QA pairs) based on the text data and the generated summaries. Specifically, it extracts questions using a question generation model, obtains appropriate answers using an information retrieval algorithm, and stores the QA pairs in a database.

[0158] Input: Text data, summary

[0159] Output: QA pair

[0160] Step 5:

[0161] Prediction of next discussion topic

[0162] The server uses a machine learning model (e.g., LSTM) to learn from past meeting records and predict the next discussion topic. Specifically, it inputs the past meeting record dataset into the model and predicts the next topic to be discussed. The prediction results are stored in a database.

[0163] Input: Past meeting recording data

[0164] Output: Next discussion topic prediction

[0165] Step 6:

[0166] Viewing the Meeting Summary

[0167] When a user accesses the conferencing tool from a device, the latest meeting summary is sent from the server to the device and displayed. Specifically, the device sends an API request to the server, and the server returns the summary data. The device then displays the received summary data to the user.

[0168] Input: API request

[0169] Output: Meeting summary

[0170] Step 7:

[0171] QA search function provided

[0172] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. Specifically, the device sends a search query to the server, which returns the best answer, and the device displays the received answer to the user.

[0173] Input: Question or keyword

[0174] Output: Answers to QA pairs

[0175] Step 8:

[0176] Display next discussion topic

[0177] The next meeting preparation screen displays the next discussion topic obtained from the server to the user. Specifically, the server sends the discussion topic to the terminal based on the prediction model, and the terminal displays this information to the user.

[0178] Input: Predictive model results

[0179] Output: Next discussion topic

[0180] In this way, by combining each processing step, the present invention can efficiently perform processes from collecting meeting records to generating summaries, generating and searching Q&A pairs, and predicting and displaying the next discussion topic.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] When operating factory robots, it is necessary to quickly identify the cause and take measures when a problem occurs. However, analyzing operation records and audio logs takes time, making it difficult to respond efficiently. In addition, proper preparations are often not made for the next maintenance or adjustment work. This leads to issues such as reduced productivity throughout the factory and increased operating costs.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for collecting operation records in a factory and applying natural language processing to generate analysis and summaries, means for generating and storing question-and-answer pairs from operation data of automated equipment, and means for predicting the topic of the next maintenance or adjustment work. This enables rapid problem solving in the operation of factory robots and efficient preparation for the next work.

[0186] "Meeting records" are text and audio data that record the contents of a meeting.

[0187] A "natural language processing model" is an algorithm or technology that allows computers to understand and analyze human language.

[0188] A summary is a short summary of the main points of a longer piece of text or record.

[0189] A "question and answer pair" is a combination of appropriate answers that correspond to a particular question.

[0190] A "database" is a system for organizing and storing information.

[0191] "Topics to be discussed next" are the main points of discussion at the next meeting predicted based on past meeting records.

[0192] A "user terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[0193] "Factory operation records" are data recorded about the operation of factory robots and equipment.

[0194] "Automation equipment" refers to robots and machines that perform tasks automatically within a factory.

[0195] "Maintenance and adjustment work" refers to the maintenance and correction work carried out to ensure that factory robots and equipment operate normally.

[0196] The present invention relates to a system for improving the operational efficiency of factory robots. Specific embodiments will be described below.

[0197] Server-side processing

[0198] First, as a means of collecting operational records within the factory, each factory robot is equipped with multiple sensors and cameras. This data is sent to a central server. The sensors used include vibration sensors and temperature sensors, and the cameras are optical sensors for video recording.

[0199] The server uses speech recognition software (e.g., the SpeechRecognition library) to convert the audio data into text. After converting the audio file to text, it analyzes it using a natural language processing (NLP) model (e.g., the Transformers library) to generate a summary of the operational recording.

[0200] Next, the server generates question and answer pairs (QA pairs) from the generated summaries and the original operational records. In this process, a question generation algorithm and an answer extraction algorithm are used. The generated QA pairs are stored in a database.

[0201] Furthermore, the server predicts topics that will be required for the next maintenance or adjustment work based on past operation records. Here, machine learning models are trained and predictions are made. The information generated in this way is sent to the factory manager's user terminal and the factory robot itself.

[0202] Terminal side processing

[0203] The user terminal displays a summary of the latest operation records sent from the server, allowing the factory manager to quickly grasp the situation.

[0204] Additionally, when a user enters a specific question or keyword, the device searches the Q&A database on the server and displays related Q&A pairs, allowing users to instantly obtain the information they need during operation or preparation.

[0205] Furthermore, on the screen for preparing for the next maintenance or adjustment work, the user's device displays the next topic obtained from the server's predictive model, allowing the user to efficiently prepare for the next work.

[0206] User processing

[0207] Factory managers can check the summary of operation records and quickly understand the details of past work. This summary function is particularly useful in emergencies and when problems occur, when quick decision-making is required.

[0208] Users can also use the Q&A search function to instantly respond to any questions or problems that arise during operation. For example, if you enter "What is the cause of the robot's picking error?", the result will be "Due to misalignment of the sensor."

[0209] A specific example would be a case where a robot frequently experiences errors during picking operations. When this happens, sensor data and audio logs are recorded and analyzed. At the next maintenance, a predictive model will indicate that "sensor calibration" is required, allowing for advance preparation.

[0210] Prompt Sentence Examples

[0211] "Please summarize the causes and solutions for errors in robot picking operations."

[0212] "Generate QA pairs regarding picking errors."

[0213] In this way, the present invention significantly improves the operational efficiency of factory robots, enabling faster problem solving and more efficient preparation for the next job.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] Data collection from factory robots

[0217] The server collects operational data from the sensors and cameras installed on each robot in the factory. Inputs include data from vibration sensors, temperature sensors, and cameras. This data is sent to the server and stored. Data processing involves standardizing the numerical data from each sensor and extracting frames from camera footage. The output is standardized data and video frames that are stored on the server as an operational record.

[0218] Step 2:

[0219] Converting audio data to text

[0220] The server converts the collected voice data into text. This process uses voice recognition software. The input is an audio file, and the output is the operation log converted into text. The audio file is imported, and the voice recognition software analyzes it and transcribes the operation log.

[0221] Step 3:

[0222] Summary generation using natural language processing

[0223] The server analyzes the converted operational records using a natural language processing (NLP) model to generate a summary. The input is the text record, and the output is the summarized text. The NLP model extracts important topics and keywords and generates a shortened summary text based on them.

[0224] Step 4:

[0225] Generate question and answer pairs

[0226] The server generates question and answer pairs (QA pairs) based on the generated summary and the original operation record. The input is the summary text and the operation record, and the output is a list of QA pairs. The QA generation algorithm analyzes the conversation flow and extracts and pairs appropriate questions and answers.

[0227] Step 5:

[0228] QA pair database storage

[0229] The server stores the generated QA pairs in a database. The input is a list of QA pairs and the output is the records stored in the database. A data insertion algorithm adds the QA pairs to the database and generates an index.

[0230] Step 6:

[0231] Prediction of upcoming maintenance topics

[0232] The server uses a machine learning model to predict topics required for the next maintenance or adjustment work from past operation records. The input is past operation records, and the output is a list of next maintenance topics. The machine learning model learns trends from past data and predicts the information required for future maintenance.

[0233] Step 7:

[0234] Summary display on user terminal

[0235] The terminal displays a summary of the latest operational records sent from the server. The input is the summary text sent from the server, and the output is the summary information displayed on the screen of the user terminal. The user interface software receives the summary and displays it in an appropriate format.

[0236] Step 8:

[0237] Execute the QA search function

[0238] When a user inputs a specific question or keyword, the terminal searches the server's QA database and displays related QA pairs. The input is the user's question text, and the output is the answer text retrieved from the QA database. The search algorithm extracts and displays the QA pairs corresponding to the question from the database.

[0239] Step 9:

[0240] Display next maintenance topic

[0241] The terminal displays the topics for the next maintenance or adjustment work obtained from the server's prediction model to the user. The input is a list of predicted maintenance topics, and the output is topic information displayed on the screen of the user's terminal. The user interface receives the topic information and displays it to help prepare for the next time.

[0242] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0243] The present invention is characterized by combining an emotion engine with a system aimed at improving meeting productivity, thereby providing optimal information according to the user's emotions. Specific embodiments of the system will be described below.

[0244] Server-side processing

[0245] 1. Collection of meeting records

[0246] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[0247] Examples:

[0248] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[0249] 2. Meeting Record Analysis and Summary Generation

[0250] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[0251] Examples:

[0252] The audio file is converted into text and key discussion points are extracted, resulting in a summary of the main topics being "budget cuts" and "marketing strategies for new products."

[0253] 3. Emotion evaluation using an emotion engine

[0254] When analyzing the meeting records, the server uses an emotion engine to evaluate the emotions of the participants, allowing it to grasp the emotional tendencies of each speaker and evaluate the flow and atmosphere of the discussion.

[0255] Examples:

[0256] The server evaluates speaker A's voice tone and choice of words during the meeting as "Speaker A has negative emotions" and "Speaker B has positive emotions."

[0257] 4. Building a QA database

[0258] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[0259] Examples:

[0260] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[0261] 5. Predicting the next discussion topic

[0262] The server uses past meeting records, generated summaries, and sentiment data to predict topics that may be discussed in the next meeting. This prediction involves using machine learning models to learn past patterns and infer future topics.

[0263] Examples:

[0264] By learning from past discussion patterns and sentiment data, it is possible to predict that the next meeting will feature a discussion on "increasing online advertising budgets."

[0265] Terminal side processing

[0266] 1. View the meeting summary

[0267] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[0268] Examples:

[0269] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[0270] 2. Displaying information according to user emotions

[0271] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be configured to simplify the display content.

[0272] Examples:

[0273] If the user is rated as "stressed," the summary will be further abridged to show only the main points.

[0274] 3. Q&A search function

[0275] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[0276] Examples:

[0277] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[0278] 4. Display next discussion topic

[0279] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[0280] Examples:

[0281] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[0282] User processing

[0283] 1. Check the summary

[0284] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[0285] Examples:

[0286] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[0287] 2. Using Q&A Search

[0288] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[0289] Examples:

[0290] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[0291] 3. Confirm the next discussion topic

[0292] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[0293] Examples:

[0294] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[0295] The present invention allows each participant to quickly obtain the information they need in real time, and makes it possible to provide optimal information according to the user's emotions, thereby significantly improving the productivity of the conference.

[0296] The processing flow will be explained below.

[0297] Step 1:

[0298] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing systems used by the company (for example, Zoom or Microsoft Teams) using APIs. As a result, the data is accumulated on the server as soon as the meeting ends.

[0299] Step 2:

[0300] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[0301] Step 3:

[0302] The server applies natural language processing (NLP) models to analyze the converted text data. Specifically, it uses IBM Watson's NLP model to analyze the meeting content in order to extract key discussion points and keywords within the text.

[0303] Step 4:

[0304] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main points of discussion and conclusions. For example, a summary such as "budget cuts and new product marketing strategies" is generated.

[0305] Step 5:

[0306] While analyzing the meeting recording, the server uses an emotion engine to assess participants' emotions, specifically, rating speaker A as having a negative emotion and speaker B as having a positive emotion based on their voice tone and word choice.

[0307] Step 6:

[0308] The server generates question and answer pairs (QA pairs) based on the generated summary and the meeting transcript. This process involves using natural language generation (NLG) technology to generate answers to potential questions.

[0309] Step 7:

[0310] The server stores the generated Q&A pairs in a database, which allows the server to quickly provide answers when users ask questions.

[0311] Step 8:

[0312] The server predicts topics that are likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. For prediction, it uses a machine learning model to learn past patterns and infer future topics.

[0313] Step 9:

[0314] When a user receives a notification of an upcoming meeting, the device accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[0315] Step 10:

[0316] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be set to display a simpler version of the information.

[0317] Step 11:

[0318] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays those answers to the user.

[0319] Step 12:

[0320] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[0321] Step 13:

[0322] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[0323] This system allows users to quickly obtain necessary information in real time and provides optimal information according to their emotions, significantly improving the productivity of meetings.

[0324] Example 2

[0325] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0326] Conventional conference systems provide insufficient information to improve conference productivity, making it difficult for users to quickly understand the content of the conference and efficiently prepare for it. Furthermore, it is difficult to provide optimal information according to the emotions of participants in the conference, resulting in a lack of improvement in the overall quality of the conference.

[0327] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records and emotion data, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, and means for displaying information optimal for the user based on emotion data. This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the conference.

[0328] "Meeting records" refer to data that records the contents of a meeting as text, audio data, or the like.

[0329] A "natural language processing model" is a model for processing natural language such as text and speech using a computer, and is used to analyze meeting records.

[0330] A "summary" is a concise summary of the main points and conclusions of a meeting.

[0331] A "question-answer pair" is a set of questions and their corresponding answers extracted from the conference transcripts and summaries.

[0332] A "database" is a structured collection of data that organizes and stores information and makes it searchable and browsable.

[0333] "Emotional data" refers to emotional information analyzed from the statements and attitudes of meeting participants.

[0334] A "machine learning model" is a computer program that learns patterns from data and uses that knowledge to make predictions and classifications.

[0335] A "user terminal" is a device used by a user to access the system, and includes a PC, tablet, smartphone, etc.

[0336] "Optimal information" refers to necessary and appropriate information provided according to the user's situation and emotions.

[0337] "Past meeting records" refers to recorded data from previous meetings, which are used to predict the next discussion topic.

[0338] "Topics to be discussed next" refers to subjects or themes that are likely to be discussed at the next meeting.

[0339] The present invention aims to improve the productivity of meetings by providing optimal information according to the user's emotions using a system that combines an emotion engine. As a specific embodiment of this system, the roles of the server, terminal, and user are described in detail below.

[0340] Server-side processing

[0341] The server collects meeting records, analyzes their contents, generates summaries, and creates question-and-answer pairs (QA pairs) that are stored in a database. It also predicts the next topic to be discussed based on past meeting records and emotion data, and provides optimal information according to the user's emotions.

[0342] Hardware and Software:

[0343] Meeting recording collection: Use Zoom API, Microsoft Teams API, etc.

[0344] Natural Language Processing: Uses the Google Cloud Natural Language API and the BERT model.

[0345] Voice data analysis: Uses Google Cloud Speech-to-Text.

[0346] Sentiment analysis: Uses IBM Watson Tone Analyzer.

[0347] Machine learning model: TENSORFLOW (registered trademark) is used.

[0348] Examples:

[0349] The server uses the Zoom API to collect meeting audio data and text transcripts, then converts the audio to text using Google Cloud Speech-to-Text. It then analyzes the transcripts using a natural language processing model (e.g., the BERT model) to generate a summary with key discussion points such as "budget cuts" and "new product marketing strategies." It then uses IBM Watson Tone Analyzer to evaluate the speaker's emotions and saves the results in a database. Next, it uses a machine learning model (e.g., TensorFlow) to learn from past meeting recordings and emotion data and predict whether "increasing the online advertising budget" is a topic that may be discussed in the next meeting.

[0350] Terminal side processing

[0351] The terminal provides an interface for users to view the latest meeting summary, emotion evaluation data, Q&A pairs, and upcoming discussion topics provided by the server, and adjusts the displayed information according to the user's emotion.

[0352] Examples:

[0353] When a user logs in to a device, the latest meeting summary is retrieved from the server and displayed. For example, a summary such as "Budget cuts, marketing strategy" may be displayed. If the user is rated as feeling stressed, the display content is further simplified to show only the main points. If a user uses the Q&A search function to ask, "When is the new product being released?", the answer displayed is "It is scheduled for March 2024." As the next meeting date approaches, the device displays an alert such as, "Next meeting topic: Online advertising budget."

[0354] User processing

[0355] Users can view meeting summaries, use the Q&A search function, check upcoming discussion topics, and instantly retrieve important information to respond to emergencies or unexpected situations.

[0356] Examples:

[0357] If a user joins a meeting late, they can check the summary in advance to understand the main points of discussion and quickly get on board with the flow of the meeting. Also, during a meeting, they can use the Q&A search function to search for a question such as "Why are there budget cuts?" and instantly get the answer "Due to rising production costs." By checking the next discussion topic in advance and preparing materials based on it, meetings can proceed smoothly.

[0358] This system allows each participant to quickly obtain the information they need in real time and provides optimal information according to the user's emotions, thereby significantly improving the productivity of meetings.

[0359] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0360] Server-side processing

[0361] Step 1: Collect meeting notes

[0362] Input: Conference system API access key and conference information

[0363] Specific behavior:

[0364] The server accesses the API of the conferencing system used by the company (e.g., Zoom, Microsoft Teams) and authenticates the access key required to obtain the conference list.

[0365] Data processing / calculation: The server makes an API call to obtain a list of target conferences.

[0366] Output: Text transcript of the meeting and audio file

[0367] Step 2: Retrieve meeting data

[0368] Input: Retrieved conference ID and information

[0369] Specific behavior:

[0370] Based on the acquired conference ID, the server uses the API to download the text minutes and audio data of the completed conference.

[0371] Data processing / calculation: The downloaded data is processed into the required format for convenient reference.

[0372] Output: Downloaded text transcript and audio file

[0373] Step 3: Convert audio data to text

[0374] Input: Audio data file

[0375] Specific behavior:

[0376] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[0377] Data processing / calculation: Analyzes language data extracted from speech and converts it into text data.

[0378] Output: Converted text data

[0379] Step 4: Analyzing the text data

[0380] Input: Text data

[0381] Specific behavior:

[0382] The server uses a natural language processing (NLP) model (e.g., the BERT model) to analyze the text data and extract key discussion points and conclusions.

[0383] Data processing / computation: Analyze data using NLP models to extract key information.

[0384] Output: Analysis data with key discussion points and conclusions

[0385] Step 5: Generate a summary

[0386] Input: Extracted discussion points and conclusions

[0387] Specific behavior:

[0388] The server generates a concise meeting summary based on the extracted information.

[0389] Data processing / computation: Applying algorithms to synthesize extracted information and generate summaries.

[0390] Output: Generated meeting summary

[0391] Step 6: Perform sentiment analysis

[0392] Input: Text data

[0393] Specific behavior:

[0394] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze participants' emotions from the text data.

[0395] Data processing / computation: Analyze language data to assess the speaker's sentiment and store it as metadata.

[0396] Output: Emotion rating data

[0397] Step 7: Storing Emotion Data

[0398] Input: Emotion rating data

[0399] Specific behavior:

[0400] The server stores the analyzed emotion data as metadata for the meeting record.

[0401] Data processing / calculation: Formatting the data appropriately for storage and adding it to the database.

[0402] Output: Stored emotion data

[0403] Step 8: Generate question and answer pairs

[0404] Input: Meeting notes, summary data

[0405] Specific behavior:

[0406] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries.

[0407] Data processing / calculation: Extract questions and corresponding answers to form pairs.

[0408] Output: Generated QA pairs

[0409] Step 9: Register with the database

[0410] Input: QA pair

[0411] Specific behavior:

[0412] The server registers the generated QA pairs in the QA database.

[0413] Data processing / calculation: Converting data into the format required for storing it in the database.

[0414] Output: Registered QA pairs

[0415] Step 10: Learn from past data

[0416] Input: Past meeting notes and emotion data

[0417] Specific behavior:

[0418] The server uses machine learning models (e.g., TensorFlow) to train them on past meeting recordings and emotion data.

[0419] Data processing / computation: Using machine learning algorithms to train datasets and build predictive models.

[0420] Output: A trained machine learning model

[0421] Step 11: Future topic prediction

[0422] Input: A trained model

[0423] Specific behavior:

[0424] The server uses the trained model to predict topics that are likely to be discussed in the next meeting.

[0425] Data processing / calculation: Predict future topics based on past data.

[0426] Output: Predicted next discussion topic

[0427] Terminal side processing

[0428] Step 1: Stay up to date

[0429] Input: Request for Updates

[0430] Specific behavior:

[0431] The terminal sends a request to obtain the latest meeting summary from the server.

[0432] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[0433] Output: The latest meeting summary retrieved from the server

[0434] Step 2: View the summary

[0435] Input: Latest meeting summary

[0436] Specific behavior:

[0437] The acquired conference summary is displayed on the terminal.

[0438] Data processing / calculation: Converting data into a concise and easy-to-understand format for display on the UI.

[0439] Output: Displayed meeting summary

[0440] Step 3: Obtaining sentiment rating data

[0441] Input: Emotion rating data request

[0442] Specific behavior:

[0443] The terminal acquires the user's emotion evaluation data from the server.

[0444] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[0445] Output: Emotion rating data obtained from the server

[0446] Step 4: Displaying the best information

[0447] Input: Emotion rating data

[0448] Specific behavior:

[0449] The device is configured to display the most appropriate information for the user based on the emotion data.

[0450] Data processing / calculation: Adjust display priority and content based on emotional data.

[0451] Output: Displaying the most appropriate information according to the user's emotions

[0452] Step 5: Submitting a search query

[0453] Input: Questions or keywords entered by the user

[0454] Specific behavior:

[0455] When a user enters a specific question or keyword, the device sends a request to search the server's Q&A database.

[0456] Data processing / calculation: Sending a search query to the server and receiving the results.

[0457] Output: Search results retrieved from the server

[0458] Step 6: Viewing QA Pairs

[0459] Input: Search results retrieved from the server

[0460] Specific behavior:

[0461] The terminal displays the QA pairs retrieved from the server.

[0462] Data processing / calculation: Converting search results into a format that can be displayed on the UI.

[0463] Output: Display of QA pairs

[0464] Step 7: Get next topic data

[0465] Input: Next discussion topic request

[0466] Specific behavior:

[0467] The terminal obtains the next discussion topic from the server.

[0468] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[0469] Output: Next discussion topic retrieved from the server

[0470] Step 8: View discussion topics

[0471] Input: Next discussion topic data

[0472] Specific behavior:

[0473] The next discussion topic will be displayed on the device.

[0474] Data processing / calculation: Format the display appropriately and present it to the user.

[0475] Output: Show next discussion topic

[0476] User processing

[0477] Step 1: Access the summary

[0478] Input: A request from a user to view a summary

[0479] Specific behavior:

[0480] The user performs an operation on the terminal to check the summary of the conference.

[0481] Data processing / calculation: Taking summary data and converting it into a viewable format.

[0482] Output: A summary of the meeting to be displayed

[0483] Step 2: Enter a search query

[0484] Input: Question or keyword

[0485] Specific behavior:

[0486] The user enters a specific question or keyword into the device's Q&A search field.

[0487] Data processing / calculation: The input data is sent to the server as a search query.

[0488] Output: Trigger to send a search query

[0489] Step 3: Check the search results

[0490] Input: QA pairs obtained from the server

[0491] Specific behavior:

[0492] The user checks the answers displayed as search results.

[0493] Data processing / calculation: Present search results in an easy-to-understand format.

[0494] Output: Display of search results

[0495] Step 4: Confirm the next discussion topic

[0496] Input: Next discussion topic data

[0497] Specific behavior:

[0498] The user checks the next discussion topic and performs operations to prepare in advance.

[0499] Data processing / calculation: Taking topic data and converting it into a viewable format.

[0500] Output: Show next discussion topic

[0501] Step 5: Prepare your materials

[0502] Input: Next discussion topic

[0503] Specific behavior:

[0504] Users prepare the necessary materials based on the next discussion topic.

[0505] Data processing / calculation: Collect and organize materials and compile them into a format that can be used at the next meeting.

[0506] Output: Prepared materials

[0507] This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the meeting.

[0508] (Application example 2)

[0509] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0510] Conventional meeting management systems have difficulty effectively summarizing the contents of meetings and enabling participants to quickly understand the key points. Furthermore, they are unable to provide information that takes into account the emotions of participants, making the outcome of the meeting susceptible to being affected by interpersonal relationships and emotions. Furthermore, they lack the support necessary to efficiently prepare for the next meeting.

[0511] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating a summary, means for analyzing the conference records and evaluating the speaker's emotions, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records, means for displaying the generated summary to a user terminal, means for adjusting the summary content based on the user's emotional state, means for searching for and displaying an answer from a database based on a question from a user terminal, and means for displaying the next topic to be discussed on a user terminal. This allows the content of the conference to be effectively summarized and provided to participants, enabling the provision of information that takes participants' emotions into consideration, and realizing efficient preparation for the next conference.

[0512] "Meeting records" refers to audio data, video data, or text data that records what was said and what was discussed during a meeting.

[0513] "Means of collection" refers to the equipment and software used to obtain meeting records using a conference system or recording device, and send them to a server.

[0514] A "natural language processing model" refers to artificial intelligence technology that analyzes voice and text data and understands their meaning and structure.

[0515] "Summary generation method" refers to algorithms or software that extracts key points and conclusions from meeting records and summarizes them in a concise form.

[0516] "Means for assessing emotions" refers to technology that analyzes speech and tone contained in meeting records to determine the emotional state of the speaker.

[0517] A "question-answer pair" refers to a set of appropriate answers corresponding to a particular question extracted from a meeting transcript or summary.

[0518] A "database" refers to a computer system for systematically storing and managing information such as question and answer pairs.

[0519] "Predictive tools" refers to techniques and algorithms that analyze past data and patterns to predict future events and topics.

[0520] A "machine learning model" refers to an artificial intelligence technology that learns patterns from data and makes predictions and classifications.

[0521] "User terminal" refers to electronic devices used by users, such as personal computers, smartphones, and tablets.

[0522] "Adjusting means" refers to technology or software for varying the amount or format of information displayed based on the user's emotional state.

[0523] "Means for searching and displaying" refers to the technology and software for searching a database based on an inquiry from a user terminal and providing the user with appropriate information.

[0524] "Meeting content" refers to the topics and conclusions discussed at the meeting, as well as the main points made.

[0525] The present invention provides a system for improving productivity of meetings in a factory. Specific embodiments will be described below.

[0526] Server-side processing

[0527] 1. Collection of meeting records

[0528] The server automatically collects meeting records using the factory's conference system and recording devices. The collected audio and video data are sent to the server. As a specific example, the server receives audio files collected from the factory's audio conference system.

[0529] 2. Meeting Record Analysis and Summary Generation

[0530] The server analyzes the collected meeting records using a natural language processing (NLP) model, extracts key discussion points and conclusions, and generates summaries. This process uses the "speech_recognition" library, which converts audio data into text, and the "Hugging Face" library, which performs natural language processing. For example, an audio file can be converted into text and summarized as key topics such as "improving production efficiency" and "marketing strategies for new products."

[0531] 3. Emotion evaluation using an emotion engine

[0532] The server uses an emotion engine to evaluate the speaker's emotion when analyzing the meeting recording. This process uses the "textblob" library to determine the speaker's emotional state. For example, the server may evaluate "Speaker A has a negative emotion" and "Speaker B has a positive emotion."

[0533] 4. Building a QA database

[0534] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries and stores them in a database. This allows for immediate provision of appropriate answers to specific questions. For example, the question "What are some ways to improve production efficiency?" and the answer "Introduce the latest equipment" are stored in the database from the meeting records.

[0535] 5. Predicting the next discussion topic

[0536] The server predicts topics likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. This prediction involves using the machine learning model "scikit-learn" to learn past patterns and infer future topics. For example, based on past discussion patterns and sentiment data, the server predicts that the next meeting will be about "announcement of a new product."

[0537] Terminal side processing

[0538] 1. View the meeting summary

[0539] When a user accesses the system from a terminal, the latest meeting summary is sent from the server to the terminal and displayed. This allows users who join a meeting late to immediately understand the main discussion content. For example, when a user receives a meeting notification, a three-line summary of last week's meeting content ("Improving production efficiency, marketing strategies") is displayed.

[0540] 2. Displaying information according to emotions

[0541] The device displays information that is optimal for the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the display content is adjusted to be simpler. As a specific example, if the user is evaluated as "feeling stressed," the summary content is further simplified and only the main points are displayed.

[0542] 3. Q&A search function

[0543] When a user enters a specific question or keyword, the device searches the server's QA database and displays related QA pairs. This allows users to instantly obtain the information they need during meetings or in preparation. For example, if a user enters "When is the new product released?", the device will display "It is scheduled for March 2024."

[0544] 4. Display next discussion topic

[0545] On the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's predictive model to the user. This allows the user to efficiently prepare for the next meeting. For example, as the next meeting date approaches, the device displays an alert such as "Next meeting topic: New product announcement."

[0546] User processing

[0547] 1. Check the summary

[0548] Users can check the summary before a meeting to quickly understand the content of past meetings. This is particularly useful when joining a meeting late or when prior preparation is insufficient. For example, even if a user joins a meeting late, they can quickly read the meeting summary to confirm the main points of discussion and quickly adapt to the flow of the meeting.

[0549] 2. Using Q&A Search

[0550] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need. This allows for more efficient discussions. For example, if you search for the question "How can we improve production efficiency?" during a meeting, the answer "By introducing the latest equipment" will immediately appear.

[0551] 3. Confirm the next discussion topic

[0552] Users can check the next discussion topic and prepare in advance more efficiently, which helps meetings proceed smoothly. For example, preparing materials for the "new product launch" in advance for the next meeting makes meeting preparations go more smoothly.

[0553] Prompt Sentence Examples

[0554] "Predict what topics will be discussed at the next meeting."

[0555] "Please summarize the meeting transcript below:

[0556] We will discuss ways to improve the factory production line. Resource optimization is necessary to increase production efficiency. At the next meeting, we will discuss the launch of a new product.

[0557] This allows each participant to quickly obtain the information they need in real time, making it possible to provide optimal information according to the user's emotions, significantly improving the productivity of the meeting.

[0558] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0559] Step 1:

[0560] The server collects the meeting records by acquiring audio and video data from the factory's conference system and recording devices, and then transmits the collected data to the server.

[0561] Input: Audio and video data from the factory's conference system and recording equipment

[0562] Data processing: Standardize the data format and transfer it to the server

[0563] Output: Meeting recording data saved on the server

[0564] Step 2:

[0565] The server converts the collected voice data into text using a natural language processing (NLP) model, using the "speech_recognition" library.

[0566] Input: Audio data stored on the server

[0567] Data processing: The process of converting voice data into text.

[0568] Output: Text transcript of the meeting

[0569] Step 3:

[0570] The server analyzes the transcribed meeting notes using a "transformers" (Hugging Face) library, extracts key discussion points and conclusions, and generates a summary.

[0571] Input: Transcribed meeting transcript

[0572] Data Computing: Using natural language processing models to extract key discussion points and generate summaries

[0573] Output: Abridged meeting transcript

[0574] Step 4:

[0575] The server analyzes the transcripts of the meeting using the "textblob" library and evaluates the emotions of each speaker. This evaluation allows the emotional tendencies of each speaker to be understood.

[0576] Input: Transcribed meeting transcript

[0577] Data calculation: Analyze text data using an emotion evaluation model to determine the emotional state

[0578] Output: Emotion rating data

[0579] Step 5:

[0580] The server generates question and answer pairs (QA pairs) from the generated summaries and the converted meeting records, and stores them in a database.

[0581] Input: Translated meeting notes, summaries

[0582] Data computation: The process of generating questions and potential answers

[0583] Output: QA pair data is saved in the database

[0584] Step 6:

[0585] Based on past meeting records and sentiment data, the server uses a machine learning model with "scikit-learn" to predict topics that are likely to be discussed at the next meeting.

[0586] Input: Past meeting records, emotional evaluation data

[0587] Data Computing: Predicting Future Discussion Topics with Machine Learning Models

[0588] Output: Predicted next discussion topic

[0589] Step 7:

[0590] The terminal receives access from the user, acquires the latest conference summary from the server, and displays it on the user terminal.

[0591] Input: Meeting summary data from the server

[0592] Data processing: Displaying summary data in a user-friendly format

[0593] Output: Meeting summary displayed on user's device

[0594] Step 8:

[0595] The terminal adjusts the display content based on the user's emotion evaluation data and provides the user with optimal information.

[0596] Input: Emotional evaluation data, meeting summary

[0597] Data processing: Adjusting the format of information according to the user's emotional state

[0598] Output: The adjusted meeting summary is displayed on the user's device.

[0599] Step 9:

[0600] The terminal receives specific questions or keywords from the user, searches the server's QA database, and displays related QA pairs.

[0601] Input: User questions or keywords

[0602] Data calculation: Database search

[0603] Output: The search results, QA pairs, are displayed on the user's device.

[0604] Step 10:

[0605] The terminal displays the next discussion topic acquired from the server to the user on the next meeting preparation screen.

[0606] Input: Next discussion topic data from the server

[0607] Data processing: Presenting data in a user-friendly format

[0608] Output: Next discussion topic displayed on the user's terminal

[0609] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0610] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0611] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0612] [Second embodiment]

[0613] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0614] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0615] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0616] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0617] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0618] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0619] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0620] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0621] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0622] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0623] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0624] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0625] The present invention is a system aimed at improving meeting productivity, and its main functions include collecting meeting records, generating summaries using natural language processing, generating question and answer pairs, building and searching a database, predicting the next discussion topic, and providing this information to user terminals.

[0626] Server-side processing

[0627] 1. Collection of meeting records

[0628] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[0629] Examples:

[0630] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[0631] 2. Meeting Record Analysis and Summary Generation

[0632] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[0633] Examples:

[0634] The audio file is converted into text, key discussion points are extracted, and a summary is generated with "budget cuts" and "marketing strategies for new products" as the main topics.

[0635] 3. Building a QA database

[0636] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[0637] Examples:

[0638] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[0639] 4. Predicting the next discussion topic

[0640] The server uses past meeting recordings to train a machine learning model to predict topics that are likely to be discussed in the next meeting, allowing users to efficiently prepare for their next meeting.

[0641] Examples:

[0642] By studying past discussion patterns, it predicts that "increasing the online advertising budget" is likely to be discussed at the next meeting.

[0643] Terminal side processing

[0644] 1. View the meeting summary

[0645] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[0646] Examples:

[0647] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[0648] 2. Q&A search function

[0649] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[0650] Examples:

[0651] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[0652] 3. Display next discussion topic

[0653] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[0654] Examples:

[0655] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[0656] User processing

[0657] 1. Check the summary

[0658] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[0659] Examples:

[0660] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[0661] 2. Using Q&A Search

[0662] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[0663] Examples:

[0664] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[0665] 3. Confirm the next discussion topic

[0666] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[0667] Examples:

[0668] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[0669] The present invention allows each participant to quickly obtain necessary information in real time, greatly improving the productivity of the meeting.

[0670] The processing flow will be explained below.

[0671] Step 1:

[0672] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing system (e.g., Zoom or Microsoft Teams). Specifically, it uses an API to retrieve the data as soon as the meeting ends.

[0673] Step 2:

[0674] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[0675] Step 3:

[0676] The server applies natural language processing (NLP) models to analyze the converted text data, which involves extracting key discussion points and keywords within the text. For example, it uses an IBM Watson NLP model to identify key discussion points.

[0677] Step 4:

[0678] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main discussion points and conclusions. For example, "budget cuts and new product marketing strategies" is summarized as the main discussion points.

[0679] Step 5:

[0680] The server generates question and answer pairs (QA pairs) based on the generated summary and meeting transcripts. This process involves using natural language generation (NLG) technology to create appropriate answers to anticipated questions.

[0681] Step 6:

[0682] The server stores the generated Q&A pairs in a database, allowing it to provide quick answers when users ask questions later.

[0683] Step 7:

[0684] The server uses past meeting recordings and generated summaries to predict topics that may be discussed in the next meeting, which involves using machine learning models to learn past patterns and infer future topics.

[0685] Step 8:

[0686] When a user receives a notification of the next meeting, the terminal accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[0687] Step 9:

[0688] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays the answers to the user.

[0689] Step 10:

[0690] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[0691] Step 11:

[0692] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[0693] Example 1

[0694] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0695] In today's business world, improving the productivity of meetings is extremely important. However, manually organizing huge amounts of meeting records, creating summaries, and predicting the next discussion topic is extremely time-consuming and labor-intensive. Another problem is the lack of a way to efficiently review meeting content and quickly retrieve information from QA databases. Therefore, a new system is needed to solve these problems and improve meeting efficiency.

[0696] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0697] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for acquiring meeting records using an API, means for converting voice data to text using a speech recognition API, means for extracting main points of the discussion and generating summaries using a natural language processing model, and means for predicting the next topic to be discussed using a machine learning model. This enables automatic collection of meeting records, generation of summaries, generation and search of Q&A pairs, and prediction of the next topic to be discussed.

[0698] "Meeting records" are data that record what was discussed, what was said, and what decisions were made at a meeting.

[0699] "Natural language processing model" refers to an algorithm or machine learning model used to understand, generate, or parse human language.

[0700] "API" stands for Application Programming Interface, a set of rules and protocols that allow software to communicate with one another.

[0701] A "speech recognition API" is an application programming interface that provides functionality for analyzing voice data and converting it into text.

[0702] "Summary generation" is the process of extracting important information from long text data and summarizing it concisely.

[0703] A "question and answer pair (QA pair)" is a combination of data that associates a suitable answer to a specific question.

[0704] A "database" is a system for systematically storing and managing large amounts of data, and for efficiently accessing, updating, and searching them.

[0705] A "machine learning model" is a collection of algorithms that learn patterns and rules based on large amounts of data and make predictions and classifications.

[0706] "User terminal" refers to a device through which a user accesses information and services, including a PC, smartphone, tablet, etc.

[0707] A "summary" is a concise expression of the main points extracted from detailed information.

[0708] "Searching" is the process of locating specific information in a database or other data source.

[0709] "Topic prediction" is the process of using past data to predict topics that are likely to be covered in future meetings or discussions.

[0710] The present invention is a system for improving the productivity of meetings, and specifically includes the following functions: collecting meeting records, generating summaries using natural language processing, generating question-and-answer pairs, building and searching a database, predicting the next topic of discussion, and providing this information to a user terminal. Below, we will explain the specific procedures for implementing the present invention, as well as the hardware and software used.

[0711] Server Processing

[0712] The server first collects meeting records from the company's conferencing system (e.g., video conferencing, audio conferencing) using an API. For example, the server uses Zoom's API to download the minutes and audio data after the meeting ends. Next, the downloaded audio data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). This text data is then analyzed using a natural language processing model (e.g., the BERT model) to extract the main points of the discussion and generate a summary.

[0713] Based on the generated summaries and meeting records, the server creates question-and-answer pairs (QA pairs). It uses a question generation model to extract questions from text data and uses information retrieval algorithms to obtain appropriate answers. The generated QA pairs are stored in a database.

[0714] Furthermore, the server uses past meeting records to train a machine learning model (e.g., LSTM) to predict the next discussion topic. This model learns past discussion patterns and predicts the topics that are likely to be discussed in the next meeting. The prediction results are also stored in the database.

[0715] Terminal handling

[0716] When a user accesses the meeting tool from their device, the server sends the latest meeting summary to the device and displays it. Specifically, the device sends an API request to the server to retrieve the summary data, and then displays this data to the user.

[0717] When a user inputs a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. The device sends a search query and the best answer is returned and displayed.

[0718] Additionally, the next meeting preparation screen displays the next discussion topic retrieved from the server by the predictive model, allowing users to efficiently prepare for the next meeting.

[0719] User Action

[0720] Users can check the summary before a meeting to quickly understand the content of past meetings. For example, even if they join a meeting late, they can quickly understand the main points discussed by checking the meeting summary.

[0721] During meetings or in preparation, users can use the Q&A search function to instantly obtain the information they need. For example, if they ask, "When is the new product released?", the answer displayed will be "It is scheduled for March 2024."

[0722] Next discussion topics are predicted using machine learning models and displayed to users, allowing them to prepare in advance for topics that are likely to be discussed in the next meeting, such as preparing materials on "online advertising budgets."

[0723] Through these procedures, the present invention can increase the efficiency of meetings and allow users to quickly obtain the information they need.

[0724] Generative AI model and example prompts

[0725] For example, the prompt to the user to get a summary for a particular conference is:

[0726] "View the summary of the weekly meeting for September 30th."

[0727] In this way, the present invention combines various technologies (APIs, speech recognition APIs, natural language processing models, and machine learning models) to automate meeting recording, summary generation, Q&A pair creation, and topic prediction, providing users with an efficient means of obtaining information.

[0728] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0729] Step 1:

[0730] Collection of meeting records

[0731] The server calls the API to retrieve the meeting records. Specifically, the server uses the API key to send a request to the endpoint of the conference system (e.g., a video conference system). The conference system's API returns the text minutes and audio data based on the conference ID. The server stores the returned data in local storage.

[0732] Input: Meeting ID, API key

[0733] Output: Text transcript, audio data

[0734] Step 2:

[0735] Analysis of meeting recordings and transcription of audio data

[0736] The server converts the acquired voice data into text data using a voice recognition API (e.g., Google Cloud Speech-to-Text). Specifically, it sends the voice data to the voice recognition API and saves the returned text data in local storage.

[0737] Input: Audio data

[0738] Output: Text data

[0739] Step 3:

[0740] Meeting Record Analysis and Summary Generation

[0741] The server uses a natural language processing model (e.g., the BERT model) to analyze the text data, extract key arguments and conclusions, and generate a summary. Specifically, the text data is input into the natural language processing model, which extracts the main points of the argument and outputs them as a summary. The server then stores the generated summary in a database.

[0742] Input: Text data

[0743] Output: Summary

[0744] Step 4:

[0745] Building a QA database

[0746] The server generates question and answer pairs (QA pairs) based on the text data and the generated summaries. Specifically, it extracts questions using a question generation model, obtains appropriate answers using an information retrieval algorithm, and stores the QA pairs in a database.

[0747] Input: Text data, summary

[0748] Output: QA pair

[0749] Step 5:

[0750] Prediction of next discussion topic

[0751] The server uses a machine learning model (e.g., LSTM) to learn from past meeting records and predict the next discussion topic. Specifically, it inputs the past meeting record dataset into the model and predicts the next topic to be discussed. The prediction results are stored in a database.

[0752] Input: Past meeting recording data

[0753] Output: Next discussion topic prediction

[0754] Step 6:

[0755] Viewing the Meeting Summary

[0756] When a user accesses the conferencing tool from a device, the latest meeting summary is sent from the server to the device and displayed. Specifically, the device sends an API request to the server, and the server returns the summary data. The device then displays the received summary data to the user.

[0757] Input: API request

[0758] Output: Meeting summary

[0759] Step 7:

[0760] QA search function provided

[0761] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. Specifically, the device sends a search query to the server, which returns the best answer, and the device displays the received answer to the user.

[0762] Input: Question or keyword

[0763] Output: Answers to QA pairs

[0764] Step 8:

[0765] Display next discussion topic

[0766] The next meeting preparation screen displays the next discussion topic obtained from the server to the user. Specifically, the server sends the discussion topic to the terminal based on the prediction model, and the terminal displays this information to the user.

[0767] Input: Predictive model results

[0768] Output: Next discussion topic

[0769] In this way, by combining each processing step, the present invention can efficiently perform processes from collecting meeting records to generating summaries, generating and searching Q&A pairs, and predicting and displaying the next discussion topic.

[0770] (Application example 1)

[0771] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0772] When operating factory robots, it is necessary to quickly identify the cause and take measures when a problem occurs. However, analyzing operation records and audio logs takes time, making it difficult to respond efficiently. In addition, proper preparations are often not made for the next maintenance or adjustment work. This leads to issues such as reduced productivity throughout the factory and increased operating costs.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0774] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for collecting operation records in a factory and applying natural language processing to generate analysis and summaries, means for generating and storing question-and-answer pairs from operation data of automated equipment, and means for predicting the topic of the next maintenance or adjustment work. This enables rapid problem solving in the operation of factory robots and efficient preparation for the next work.

[0775] "Meeting records" are text and audio data that record the contents of a meeting.

[0776] A "natural language processing model" is an algorithm or technology that allows computers to understand and analyze human language.

[0777] A summary is a short summary of the main points of a longer piece of text or record.

[0778] A "question and answer pair" is a combination of appropriate answers that correspond to a particular question.

[0779] A "database" is a system for organizing and storing information.

[0780] "Topics to be discussed next" are the main points of discussion at the next meeting predicted based on past meeting records.

[0781] A "user terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[0782] "Factory operation records" are data recorded about the operation of factory robots and equipment.

[0783] "Automation equipment" refers to robots and machines that perform tasks automatically within a factory.

[0784] "Maintenance and adjustment work" refers to the maintenance and correction work carried out to ensure that factory robots and equipment operate normally.

[0785] The present invention relates to a system for improving the operational efficiency of factory robots. Specific embodiments will be described below.

[0786] Server-side processing

[0787] First, as a means of collecting operational records within the factory, each factory robot is equipped with multiple sensors and cameras. This data is sent to a central server. The sensors used include vibration sensors and temperature sensors, and the cameras are optical sensors for video recording.

[0788] The server uses speech recognition software (e.g., the SpeechRecognition library) to convert the audio data into text. After converting the audio file to text, it analyzes it using a natural language processing (NLP) model (e.g., the Transformers library) to generate a summary of the operational recording.

[0789] Next, the server generates question and answer pairs (QA pairs) from the generated summaries and the original operational records. In this process, a question generation algorithm and an answer extraction algorithm are used. The generated QA pairs are stored in a database.

[0790] Furthermore, the server predicts topics that will be required for the next maintenance or adjustment work based on past operation records. Here, machine learning models are trained and predictions are made. The information generated in this way is sent to the factory manager's user terminal and the factory robot itself.

[0791] Terminal side processing

[0792] The user terminal displays a summary of the latest operation records sent from the server, allowing the factory manager to quickly grasp the situation.

[0793] Additionally, when a user enters a specific question or keyword, the device searches the Q&A database on the server and displays related Q&A pairs, allowing users to instantly obtain the information they need during operation or preparation.

[0794] Furthermore, on the screen for preparing for the next maintenance or adjustment work, the user's device displays the next topic obtained from the server's predictive model, allowing the user to efficiently prepare for the next work.

[0795] User processing

[0796] Factory managers can check the summary of operation records and quickly understand the details of past work. This summary function is particularly useful in emergencies and when problems occur, when quick decision-making is required.

[0797] Users can also use the Q&A search function to instantly respond to any questions or problems that arise during operation. For example, if you enter "What is the cause of the robot's picking error?", the result will be "Due to misalignment of the sensor."

[0798] A specific example would be a case where a robot frequently experiences errors during picking operations. When this happens, sensor data and audio logs are recorded and analyzed. At the next maintenance, a predictive model will indicate that "sensor calibration" is required, allowing for advance preparation.

[0799] Prompt Sentence Examples

[0800] "Please summarize the causes and solutions for errors in robot picking operations."

[0801] "Generate QA pairs regarding picking errors."

[0802] In this way, the present invention significantly improves the operational efficiency of factory robots, enabling faster problem solving and more efficient preparation for the next job.

[0803] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0804] Step 1:

[0805] Data collection from factory robots

[0806] The server collects operational data from the sensors and cameras installed on each robot in the factory. Inputs include data from vibration sensors, temperature sensors, and cameras. This data is sent to the server and stored. Data processing involves standardizing the numerical data from each sensor and extracting frames from camera footage. The output is standardized data and video frames that are stored on the server as an operational record.

[0807] Step 2:

[0808] Converting audio data to text

[0809] The server converts the collected voice data into text. This process uses voice recognition software. The input is an audio file, and the output is the operation log converted into text. The audio file is imported, and the voice recognition software analyzes it and transcribes the operation log.

[0810] Step 3:

[0811] Summary generation using natural language processing

[0812] The server analyzes the converted operational records using a natural language processing (NLP) model to generate a summary. The input is the text record, and the output is the summarized text. The NLP model extracts important topics and keywords and generates a shortened summary text based on them.

[0813] Step 4:

[0814] Generate question and answer pairs

[0815] The server generates question and answer pairs (QA pairs) based on the generated summary and the original operation record. The input is the summary text and the operation record, and the output is a list of QA pairs. The QA generation algorithm analyzes the conversation flow and extracts and pairs appropriate questions and answers.

[0816] Step 5:

[0817] QA pair database storage

[0818] The server stores the generated QA pairs in a database. The input is a list of QA pairs and the output is the records stored in the database. A data insertion algorithm adds the QA pairs to the database and generates an index.

[0819] Step 6:

[0820] Prediction of upcoming maintenance topics

[0821] The server uses a machine learning model to predict topics required for the next maintenance or adjustment work from past operation records. The input is past operation records, and the output is a list of next maintenance topics. The machine learning model learns trends from past data and predicts the information required for future maintenance.

[0822] Step 7:

[0823] Summary display on user terminal

[0824] The terminal displays a summary of the latest operational records sent from the server. The input is the summary text sent from the server, and the output is the summary information displayed on the screen of the user terminal. The user interface software receives the summary and displays it in an appropriate format.

[0825] Step 8:

[0826] Execute the QA search function

[0827] When a user inputs a specific question or keyword, the terminal searches the server's QA database and displays related QA pairs. The input is the user's question text, and the output is the answer text retrieved from the QA database. The search algorithm extracts and displays the QA pairs corresponding to the question from the database.

[0828] Step 9:

[0829] Display next maintenance topic

[0830] The terminal displays the topics for the next maintenance or adjustment work obtained from the server's prediction model to the user. The input is a list of predicted maintenance topics, and the output is topic information displayed on the screen of the user's terminal. The user interface receives the topic information and displays it to help prepare for the next time.

[0831] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0832] The present invention is characterized by combining an emotion engine with a system aimed at improving meeting productivity, thereby providing optimal information according to the user's emotions. Specific embodiments of the system will be described below.

[0833] Server-side processing

[0834] 1. Collection of meeting records

[0835] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[0836] Examples:

[0837] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[0838] 2. Meeting Record Analysis and Summary Generation

[0839] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[0840] Examples:

[0841] The audio file is converted into text and key discussion points are extracted, resulting in a summary of the main topics being "budget cuts" and "marketing strategies for new products."

[0842] 3. Emotion evaluation using an emotion engine

[0843] When analyzing the meeting records, the server uses an emotion engine to evaluate the emotions of the participants, allowing it to grasp the emotional tendencies of each speaker and evaluate the flow and atmosphere of the discussion.

[0844] Examples:

[0845] The server evaluates speaker A's voice tone and choice of words during the meeting as "Speaker A has negative emotions" and "Speaker B has positive emotions."

[0846] 4. Building a QA database

[0847] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[0848] Examples:

[0849] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[0850] 5. Predicting the next discussion topic

[0851] The server uses past meeting records, generated summaries, and sentiment data to predict topics that may be discussed in the next meeting. This prediction involves using machine learning models to learn past patterns and infer future topics.

[0852] Examples:

[0853] By learning from past discussion patterns and sentiment data, it is possible to predict that the next meeting will feature a discussion on "increasing online advertising budgets."

[0854] Terminal side processing

[0855] 1. View the meeting summary

[0856] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[0857] Examples:

[0858] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[0859] 2. Displaying information according to user emotions

[0860] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be configured to simplify the display content.

[0861] Examples:

[0862] If the user is rated as "stressed," the summary will be further abridged to show only the main points.

[0863] 3. Q&A search function

[0864] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[0865] Examples:

[0866] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[0867] 4. Display next discussion topic

[0868] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[0869] Examples:

[0870] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[0871] User processing

[0872] 1. Check the summary

[0873] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[0874] Examples:

[0875] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[0876] 2. Using Q&A Search

[0877] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[0878] Examples:

[0879] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[0880] 3. Confirm the next discussion topic

[0881] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[0882] Examples:

[0883] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[0884] The present invention allows each participant to quickly obtain the information they need in real time, and makes it possible to provide optimal information according to the user's emotions, thereby significantly improving the productivity of the conference.

[0885] The processing flow will be explained below.

[0886] Step 1:

[0887] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing systems used by the company (for example, Zoom or Microsoft Teams) using APIs. As a result, the data is accumulated on the server as soon as the meeting ends.

[0888] Step 2:

[0889] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[0890] Step 3:

[0891] The server applies natural language processing (NLP) models to analyze the converted text data. Specifically, it uses IBM Watson's NLP model to analyze the meeting content in order to extract key discussion points and keywords within the text.

[0892] Step 4:

[0893] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main points of discussion and conclusions. For example, a summary such as "budget cuts and new product marketing strategies" is generated.

[0894] Step 5:

[0895] While analyzing the meeting recording, the server uses an emotion engine to assess participants' emotions, specifically, rating speaker A as having a negative emotion and speaker B as having a positive emotion based on their voice tone and word choice.

[0896] Step 6:

[0897] The server generates question and answer pairs (QA pairs) based on the generated summary and the meeting transcript. This process involves using natural language generation (NLG) technology to generate answers to potential questions.

[0898] Step 7:

[0899] The server stores the generated Q&A pairs in a database, which allows the server to quickly provide answers when users ask questions.

[0900] Step 8:

[0901] The server predicts topics that are likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. For prediction, it uses a machine learning model to learn past patterns and infer future topics.

[0902] Step 9:

[0903] When a user receives a notification of an upcoming meeting, the device accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[0904] Step 10:

[0905] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be set to display a simpler version of the information.

[0906] Step 11:

[0907] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays those answers to the user.

[0908] Step 12:

[0909] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[0910] Step 13:

[0911] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[0912] This system allows users to quickly obtain necessary information in real time and provides optimal information according to their emotions, significantly improving the productivity of meetings.

[0913] Example 2

[0914] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0915] Conventional conference systems provide insufficient information to improve conference productivity, making it difficult for users to quickly understand the content of the conference and efficiently prepare for it. Furthermore, it is difficult to provide optimal information according to the emotions of participants in the conference, resulting in a lack of improvement in the overall quality of the conference.

[0916] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records and emotion data, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, and means for displaying information optimal for the user based on emotion data. This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the conference.

[0917] "Meeting records" refer to data that records the contents of a meeting as text, audio data, or the like.

[0918] A "natural language processing model" is a model for processing natural language such as text and speech using a computer, and is used to analyze meeting records.

[0919] A "summary" is a concise summary of the main points and conclusions of a meeting.

[0920] A "question-answer pair" is a set of questions and their corresponding answers extracted from the conference transcripts and summaries.

[0921] A "database" is a structured collection of data that organizes and stores information and makes it searchable and browsable.

[0922] "Emotional data" refers to emotional information analyzed from the statements and attitudes of meeting participants.

[0923] A "machine learning model" is a computer program that learns patterns from data and uses that knowledge to make predictions and classifications.

[0924] A "user terminal" is a device used by a user to access the system, and includes a PC, tablet, smartphone, etc.

[0925] "Optimal information" refers to necessary and appropriate information provided according to the user's situation and emotions.

[0926] "Past meeting records" refers to recorded data from previous meetings, which are used to predict the next discussion topic.

[0927] "Topics to be discussed next" refers to subjects or themes that are likely to be discussed at the next meeting.

[0928] The present invention aims to improve the productivity of meetings by providing optimal information according to the user's emotions using a system that combines an emotion engine. As a specific embodiment of this system, the roles of the server, terminal, and user are described in detail below.

[0929] Server-side processing

[0930] The server collects meeting records, analyzes their contents, generates summaries, and creates question-and-answer pairs (QA pairs) that are stored in a database. It also predicts the next topic to be discussed based on past meeting records and emotion data, and provides optimal information according to the user's emotions.

[0931] Hardware and Software:

[0932] Meeting recording collection: Use Zoom API, Microsoft Teams API, etc.

[0933] Natural Language Processing: Uses the Google Cloud Natural Language API and the BERT model.

[0934] Voice data analysis: Uses Google Cloud Speech-to-Text.

[0935] Sentiment analysis: Uses IBM Watson Tone Analyzer.

[0936] Machine learning model: Uses TensorFlow.

[0937] Examples:

[0938] The server uses the Zoom API to collect meeting audio data and text transcripts, then converts the audio to text using Google Cloud Speech-to-Text. It then analyzes the transcripts using a natural language processing model (e.g., the BERT model) to generate a summary with key discussion points such as "budget cuts" and "new product marketing strategies." It then uses IBM Watson Tone Analyzer to evaluate the speaker's emotions and saves the results in a database. Next, it uses a machine learning model (e.g., TensorFlow) to learn from past meeting recordings and emotion data and predict whether "increasing the online advertising budget" is a topic that may be discussed in the next meeting.

[0939] Terminal side processing

[0940] The terminal provides an interface for users to view the latest meeting summary, emotion evaluation data, Q&A pairs, and upcoming discussion topics provided by the server, and adjusts the displayed information according to the user's emotion.

[0941] Examples:

[0942] When a user logs in to a device, the latest meeting summary is retrieved from the server and displayed. For example, a summary such as "Budget cuts, marketing strategy" may be displayed. If the user is rated as feeling stressed, the display content is further simplified to show only the main points. If a user uses the Q&A search function to ask, "When is the new product being released?", the answer displayed is "It is scheduled for March 2024." As the next meeting date approaches, the device displays an alert such as, "Next meeting topic: Online advertising budget."

[0943] User processing

[0944] Users can view meeting summaries, use the Q&A search function, check upcoming discussion topics, and instantly retrieve important information to respond to emergencies or unexpected situations.

[0945] Examples:

[0946] If a user joins a meeting late, they can check the summary in advance to understand the main points of discussion and quickly get on board with the flow of the meeting. Also, during a meeting, they can use the Q&A search function to search for a question such as "Why are there budget cuts?" and instantly get the answer "Due to rising production costs." By checking the next discussion topic in advance and preparing materials based on it, meetings can proceed smoothly.

[0947] This system allows each participant to quickly obtain the information they need in real time and provides optimal information according to the user's emotions, thereby significantly improving the productivity of meetings.

[0948] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0949] Server-side processing

[0950] Step 1: Collect meeting notes

[0951] Input: Conference system API access key and conference information

[0952] Specific behavior:

[0953] The server accesses the API of the conferencing system used by the company (e.g., Zoom, Microsoft Teams) and authenticates the access key required to obtain the conference list.

[0954] Data processing / calculation: The server makes an API call to obtain a list of target conferences.

[0955] Output: Text transcript of the meeting and audio file

[0956] Step 2: Retrieve meeting data

[0957] Input: Retrieved conference ID and information

[0958] Specific behavior:

[0959] Based on the acquired conference ID, the server uses the API to download the text minutes and audio data of the completed conference.

[0960] Data processing / calculation: The downloaded data is processed into the required format for convenient reference.

[0961] Output: Downloaded text transcript and audio file

[0962] Step 3: Convert audio data to text

[0963] Input: Audio data file

[0964] Specific behavior:

[0965] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[0966] Data processing / calculation: Analyzes language data extracted from speech and converts it into text data.

[0967] Output: Converted text data

[0968] Step 4: Analyzing the text data

[0969] Input: Text data

[0970] Specific behavior:

[0971] The server uses a natural language processing (NLP) model (e.g., the BERT model) to analyze the text data and extract key discussion points and conclusions.

[0972] Data processing / computation: Analyze data using NLP models to extract key information.

[0973] Output: Analysis data with key discussion points and conclusions

[0974] Step 5: Generate a summary

[0975] Input: Extracted discussion points and conclusions

[0976] Specific behavior:

[0977] The server generates a concise meeting summary based on the extracted information.

[0978] Data processing / computation: Applying algorithms to synthesize extracted information and generate summaries.

[0979] Output: Generated meeting summary

[0980] Step 6: Perform sentiment analysis

[0981] Input: Text data

[0982] Specific behavior:

[0983] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze participants' emotions from the text data.

[0984] Data processing / computation: Analyze language data to assess the speaker's sentiment and store it as metadata.

[0985] Output: Emotion rating data

[0986] Step 7: Storing Emotion Data

[0987] Input: Emotion rating data

[0988] Specific behavior:

[0989] The server stores the analyzed emotion data as metadata for the meeting record.

[0990] Data processing / calculation: Formatting the data appropriately for storage and adding it to the database.

[0991] Output: Stored emotion data

[0992] Step 8: Generate question and answer pairs

[0993] Input: Meeting notes, summary data

[0994] Specific behavior:

[0995] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries.

[0996] Data processing / calculation: Extract questions and corresponding answers to form pairs.

[0997] Output: Generated QA pairs

[0998] Step 9: Register with the database

[0999] Input: QA pair

[1000] Specific behavior:

[1001] The server registers the generated QA pairs in the QA database.

[1002] Data processing / calculation: Converting data into the format required for storing it in the database.

[1003] Output: Registered QA pairs

[1004] Step 10: Learn from past data

[1005] Input: Past meeting notes and emotion data

[1006] Specific behavior:

[1007] The server uses machine learning models (e.g., TensorFlow) to train them on past meeting recordings and emotion data.

[1008] Data processing / computation: Using machine learning algorithms to train datasets and build predictive models.

[1009] Output: A trained machine learning model

[1010] Step 11: Future topic prediction

[1011] Input: A trained model

[1012] Specific behavior:

[1013] The server uses the trained model to predict topics that are likely to be discussed in the next meeting.

[1014] Data processing / calculation: Predict future topics based on past data.

[1015] Output: Predicted next discussion topic

[1016] Terminal side processing

[1017] Step 1: Stay up to date

[1018] Input: Request for Updates

[1019] Specific behavior:

[1020] The terminal sends a request to obtain the latest meeting summary from the server.

[1021] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[1022] Output: The latest meeting summary retrieved from the server

[1023] Step 2: View the summary

[1024] Input: Latest meeting summary

[1025] Specific behavior:

[1026] The acquired conference summary is displayed on the terminal.

[1027] Data processing / calculation: Converting data into a concise and easy-to-understand format for display on the UI.

[1028] Output: Displayed meeting summary

[1029] Step 3: Obtaining sentiment rating data

[1030] Input: Emotion rating data request

[1031] Specific behavior:

[1032] The terminal acquires the user's emotion evaluation data from the server.

[1033] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[1034] Output: Emotion rating data obtained from the server

[1035] Step 4: Displaying the best information

[1036] Input: Emotion rating data

[1037] Specific behavior:

[1038] The device is configured to display the most appropriate information for the user based on the emotion data.

[1039] Data processing / calculation: Adjust display priority and content based on emotional data.

[1040] Output: Displaying the most appropriate information according to the user's emotions

[1041] Step 5: Submitting a search query

[1042] Input: Questions or keywords entered by the user

[1043] Specific behavior:

[1044] When a user enters a specific question or keyword, the device sends a request to search the server's Q&A database.

[1045] Data processing / calculation: Sending a search query to the server and receiving the results.

[1046] Output: Search results retrieved from the server

[1047] Step 6: Viewing QA Pairs

[1048] Input: Search results retrieved from the server

[1049] Specific behavior:

[1050] The terminal displays the QA pairs retrieved from the server.

[1051] Data processing / calculation: Converting search results into a format that can be displayed on the UI.

[1052] Output: Display of QA pairs

[1053] Step 7: Get next topic data

[1054] Input: Next discussion topic request

[1055] Specific behavior:

[1056] The terminal obtains the next discussion topic from the server.

[1057] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[1058] Output: Next discussion topic retrieved from the server

[1059] Step 8: View discussion topics

[1060] Input: Next discussion topic data

[1061] Specific behavior:

[1062] The next discussion topic will be displayed on the device.

[1063] Data processing / calculation: Format the display appropriately and present it to the user.

[1064] Output: Show next discussion topic

[1065] User processing

[1066] Step 1: Access the summary

[1067] Input: A request from a user to view a summary

[1068] Specific behavior:

[1069] The user performs an operation on the terminal to check the summary of the conference.

[1070] Data processing / calculation: Taking summary data and converting it into a viewable format.

[1071] Output: A summary of the meeting to be displayed

[1072] Step 2: Enter a search query

[1073] Input: Question or keyword

[1074] Specific behavior:

[1075] The user enters a specific question or keyword into the device's Q&A search field.

[1076] Data processing / calculation: The input data is sent to the server as a search query.

[1077] Output: Trigger to send a search query

[1078] Step 3: Check the search results

[1079] Input: QA pairs obtained from the server

[1080] Specific behavior:

[1081] The user checks the answers displayed as search results.

[1082] Data processing / calculation: Present search results in an easy-to-understand format.

[1083] Output: Display of search results

[1084] Step 4: Confirm the next discussion topic

[1085] Input: Next discussion topic data

[1086] Specific behavior:

[1087] The user checks the next discussion topic and performs operations to prepare in advance.

[1088] Data processing / calculation: Taking topic data and converting it into a viewable format.

[1089] Output: Show next discussion topic

[1090] Step 5: Prepare your materials

[1091] Input: Next discussion topic

[1092] Specific behavior:

[1093] Users prepare the necessary materials based on the next discussion topic.

[1094] Data processing / calculation: Collect and organize materials and compile them into a format that can be used at the next meeting.

[1095] Output: Prepared materials

[1096] This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the meeting.

[1097] (Application example 2)

[1098] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1099] Conventional meeting management systems have difficulty effectively summarizing the contents of meetings and enabling participants to quickly understand the key points. Furthermore, they are unable to provide information that takes into account the emotions of participants, making the outcome of the meeting susceptible to being affected by interpersonal relationships and emotions. Furthermore, they lack the support necessary to efficiently prepare for the next meeting.

[1100] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating a summary, means for analyzing the conference records and evaluating the speaker's emotions, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records, means for displaying the generated summary to a user terminal, means for adjusting the summary content based on the user's emotional state, means for searching for and displaying an answer from a database based on a question from a user terminal, and means for displaying the next topic to be discussed on a user terminal. This allows the content of the conference to be effectively summarized and provided to participants, enabling the provision of information that takes participants' emotions into consideration, and realizing efficient preparation for the next conference.

[1101] "Meeting records" refers to audio data, video data, or text data that records what was said and what was discussed during a meeting.

[1102] "Means of collection" refers to the equipment and software used to obtain meeting records using a conference system or recording device, and send them to a server.

[1103] A "natural language processing model" refers to artificial intelligence technology that analyzes voice and text data and understands their meaning and structure.

[1104] "Summary generation method" refers to algorithms or software that extracts key points and conclusions from meeting records and summarizes them in a concise form.

[1105] "Means for assessing emotions" refers to technology that analyzes speech and tone contained in meeting records to determine the emotional state of the speaker.

[1106] A "question-answer pair" refers to a set of appropriate answers corresponding to a particular question extracted from a meeting transcript or summary.

[1107] A "database" refers to a computer system for systematically storing and managing information such as question and answer pairs.

[1108] "Predictive tools" refers to techniques and algorithms that analyze past data and patterns to predict future events and topics.

[1109] A "machine learning model" refers to an artificial intelligence technology that learns patterns from data and makes predictions and classifications.

[1110] "User terminal" refers to electronic devices used by users, such as personal computers, smartphones, and tablets.

[1111] "Adjusting means" refers to technology or software for varying the amount or format of information displayed based on the user's emotional state.

[1112] "Means for searching and displaying" refers to the technology and software for searching a database based on an inquiry from a user terminal and providing the user with appropriate information.

[1113] "Meeting content" refers to the topics and conclusions discussed at the meeting, as well as the main points made.

[1114] The present invention provides a system for improving productivity of meetings in a factory. Specific embodiments will be described below.

[1115] Server-side processing

[1116] 1. Collection of meeting records

[1117] The server automatically collects meeting records using the factory's conference system and recording devices. The collected audio and video data are sent to the server. As a specific example, the server receives audio files collected from the factory's audio conference system.

[1118] 2. Meeting Record Analysis and Summary Generation

[1119] The server analyzes the collected meeting records using a natural language processing (NLP) model, extracts key discussion points and conclusions, and generates summaries. This process uses the "speech_recognition" library, which converts audio data into text, and the "Hugging Face" library, which performs natural language processing. For example, an audio file can be converted into text and summarized as key topics such as "improving production efficiency" and "marketing strategies for new products."

[1120] 3. Emotion evaluation using an emotion engine

[1121] The server uses an emotion engine to evaluate the speaker's emotion when analyzing the meeting recording. This process uses the "textblob" library to determine the speaker's emotional state. For example, the server may evaluate "Speaker A has a negative emotion" and "Speaker B has a positive emotion."

[1122] 4. Building a QA database

[1123] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries and stores them in a database. This allows for immediate provision of appropriate answers to specific questions. For example, the question "What are some ways to improve production efficiency?" and the answer "Introduce the latest equipment" are stored in the database from the meeting records.

[1124] 5. Predicting the next discussion topic

[1125] The server predicts topics likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. This prediction involves using the machine learning model "scikit-learn" to learn past patterns and infer future topics. For example, based on past discussion patterns and sentiment data, the server predicts that the next meeting will be about "announcement of a new product."

[1126] Terminal side processing

[1127] 1. View the meeting summary

[1128] When a user accesses the system from a terminal, the latest meeting summary is sent from the server to the terminal and displayed. This allows users who join a meeting late to immediately understand the main discussion content. For example, when a user receives a meeting notification, a three-line summary of last week's meeting content ("Improving production efficiency, marketing strategies") is displayed.

[1129] 2. Displaying information according to emotions

[1130] The device displays information that is optimal for the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the display content is adjusted to be simpler. As a specific example, if the user is evaluated as "feeling stressed," the summary content is further simplified and only the main points are displayed.

[1131] 3. Q&A search function

[1132] When a user enters a specific question or keyword, the device searches the server's QA database and displays related QA pairs. This allows users to instantly obtain the information they need during meetings or in preparation. For example, if a user enters "When is the new product released?", the device will display "It is scheduled for March 2024."

[1133] 4. Display next discussion topic

[1134] On the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's predictive model to the user. This allows the user to efficiently prepare for the next meeting. For example, as the next meeting date approaches, the device displays an alert such as "Next meeting topic: New product announcement."

[1135] User processing

[1136] 1. Check the summary

[1137] Users can check the summary before a meeting to quickly understand the content of past meetings. This is particularly useful when joining a meeting late or when prior preparation is insufficient. For example, even if a user joins a meeting late, they can quickly read the meeting summary to confirm the main points of discussion and quickly adapt to the flow of the meeting.

[1138] 2. Using Q&A Search

[1139] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need. This allows for more efficient discussions. For example, if you search for the question "How can we improve production efficiency?" during a meeting, the answer "By introducing the latest equipment" will immediately appear.

[1140] 3. Confirm the next discussion topic

[1141] Users can check the next discussion topic and prepare in advance more efficiently, which helps meetings proceed smoothly. For example, preparing materials for the "new product launch" in advance for the next meeting makes meeting preparations go more smoothly.

[1142] Prompt Sentence Examples

[1143] "Predict what topics will be discussed at the next meeting."

[1144] "Please summarize the meeting transcript below:

[1145] We will discuss ways to improve the factory production line. Resource optimization is necessary to increase production efficiency. At the next meeting, we will discuss the launch of a new product.

[1146] This allows each participant to quickly obtain the information they need in real time, making it possible to provide optimal information according to the user's emotions, significantly improving the productivity of the meeting.

[1147] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1148] Step 1:

[1149] The server collects the meeting records by acquiring audio and video data from the factory's conference system and recording devices, and then transmits the collected data to the server.

[1150] Input: Audio and video data from the factory's conference system and recording equipment

[1151] Data processing: Standardize the data format and transfer it to the server

[1152] Output: Meeting recording data saved on the server

[1153] Step 2:

[1154] The server converts the collected voice data into text using a natural language processing (NLP) model, using the "speech_recognition" library.

[1155] Input: Audio data stored on the server

[1156] Data processing: The process of converting voice data into text.

[1157] Output: Text transcript of the meeting

[1158] Step 3:

[1159] The server analyzes the transcribed meeting notes using a "transformers" (Hugging Face) library, extracts key discussion points and conclusions, and generates a summary.

[1160] Input: Transcribed meeting transcript

[1161] Data Computing: Using natural language processing models to extract key discussion points and generate summaries

[1162] Output: Abridged meeting transcript

[1163] Step 4:

[1164] The server analyzes the transcripts of the meeting using the "textblob" library and evaluates the emotions of each speaker. This evaluation allows the emotional tendencies of each speaker to be understood.

[1165] Input: Transcribed meeting transcript

[1166] Data calculation: Analyze text data using an emotion evaluation model to determine the emotional state

[1167] Output: Emotion rating data

[1168] Step 5:

[1169] The server generates question and answer pairs (QA pairs) from the generated summaries and the converted meeting records, and stores them in a database.

[1170] Input: Translated meeting notes, summaries

[1171] Data computation: The process of generating questions and potential answers

[1172] Output: QA pair data is saved in the database

[1173] Step 6:

[1174] Based on past meeting records and sentiment data, the server uses a machine learning model with "scikit-learn" to predict topics that are likely to be discussed at the next meeting.

[1175] Input: Past meeting records, emotional evaluation data

[1176] Data Computing: Predicting Future Discussion Topics with Machine Learning Models

[1177] Output: Predicted next discussion topic

[1178] Step 7:

[1179] The terminal receives access from the user, acquires the latest conference summary from the server, and displays it on the user terminal.

[1180] Input: Meeting summary data from the server

[1181] Data processing: Displaying summary data in a user-friendly format

[1182] Output: Meeting summary displayed on user's device

[1183] Step 8:

[1184] The terminal adjusts the display content based on the user's emotion evaluation data and provides the user with optimal information.

[1185] Input: Emotional evaluation data, meeting summary

[1186] Data processing: Adjusting the format of information according to the user's emotional state

[1187] Output: The adjusted meeting summary is displayed on the user's device.

[1188] Step 9:

[1189] The terminal receives specific questions or keywords from the user, searches the server's QA database, and displays related QA pairs.

[1190] Input: User questions or keywords

[1191] Data calculation: Database search

[1192] Output: The search results, QA pairs, are displayed on the user's device.

[1193] Step 10:

[1194] The terminal displays the next discussion topic acquired from the server to the user on the next meeting preparation screen.

[1195] Input: Next discussion topic data from the server

[1196] Data processing: Presenting data in a user-friendly format

[1197] Output: Next discussion topic displayed on the user's terminal

[1198] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1199] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1200] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1201] [Third embodiment]

[1202] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1203] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1204] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1205] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1206] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1207] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1208] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1209] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1210] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1211] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1212] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1213] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1214] The present invention is a system aimed at improving meeting productivity, and its main functions include collecting meeting records, generating summaries using natural language processing, generating question and answer pairs, building and searching a database, predicting the next discussion topic, and providing this information to user terminals.

[1215] Server-side processing

[1216] 1. Collection of meeting records

[1217] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[1218] Examples:

[1219] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[1220] 2. Meeting Record Analysis and Summary Generation

[1221] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[1222] Examples:

[1223] The audio file is converted into text, key discussion points are extracted, and a summary is generated with "budget cuts" and "marketing strategies for new products" as the main topics.

[1224] 3. Building a QA database

[1225] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[1226] Examples:

[1227] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[1228] 4. Predicting the next discussion topic

[1229] The server uses past meeting recordings to train a machine learning model to predict topics that are likely to be discussed in the next meeting, allowing users to efficiently prepare for their next meeting.

[1230] Examples:

[1231] By studying past discussion patterns, it predicts that "increasing the online advertising budget" is likely to be discussed at the next meeting.

[1232] Terminal side processing

[1233] 1. View the meeting summary

[1234] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[1235] Examples:

[1236] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[1237] 2. Q&A search function

[1238] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[1239] Examples:

[1240] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[1241] 3. Display next discussion topic

[1242] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[1243] Examples:

[1244] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[1245] User processing

[1246] 1. Check the summary

[1247] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[1248] Examples:

[1249] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[1250] 2. Using Q&A Search

[1251] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[1252] Examples:

[1253] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[1254] 3. Confirm the next discussion topic

[1255] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[1256] Examples:

[1257] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[1258] The present invention allows each participant to quickly obtain necessary information in real time, greatly improving the productivity of the meeting.

[1259] The processing flow will be explained below.

[1260] Step 1:

[1261] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing system (e.g., Zoom or Microsoft Teams). Specifically, it uses an API to retrieve the data as soon as the meeting ends.

[1262] Step 2:

[1263] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[1264] Step 3:

[1265] The server applies natural language processing (NLP) models to analyze the converted text data, which involves extracting key discussion points and keywords within the text. For example, it uses an IBM Watson NLP model to identify key discussion points.

[1266] Step 4:

[1267] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main discussion points and conclusions. For example, "budget cuts and new product marketing strategies" is summarized as the main discussion points.

[1268] Step 5:

[1269] The server generates question and answer pairs (QA pairs) based on the generated summary and meeting transcripts. This process involves using natural language generation (NLG) technology to create appropriate answers to anticipated questions.

[1270] Step 6:

[1271] The server stores the generated Q&A pairs in a database, allowing it to provide quick answers when users ask questions later.

[1272] Step 7:

[1273] The server uses past meeting recordings and generated summaries to predict topics that may be discussed in the next meeting, which involves using machine learning models to learn past patterns and infer future topics.

[1274] Step 8:

[1275] When a user receives a notification of the next meeting, the terminal accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[1276] Step 9:

[1277] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays the answers to the user.

[1278] Step 10:

[1279] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[1280] Step 11:

[1281] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[1282] Example 1

[1283] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1284] In today's business world, improving the productivity of meetings is extremely important. However, manually organizing huge amounts of meeting records, creating summaries, and predicting the next discussion topic is extremely time-consuming and labor-intensive. Another problem is the lack of a way to efficiently review meeting content and quickly retrieve information from QA databases. Therefore, a new system is needed to solve these problems and improve meeting efficiency.

[1285] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1286] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for acquiring meeting records using an API, means for converting voice data to text using a speech recognition API, means for extracting main points of the discussion and generating summaries using a natural language processing model, and means for predicting the next topic to be discussed using a machine learning model. This enables automatic collection of meeting records, generation of summaries, generation and search of Q&A pairs, and prediction of the next topic to be discussed.

[1287] "Meeting records" are data that record what was discussed, what was said, and what decisions were made at a meeting.

[1288] "Natural language processing model" refers to an algorithm or machine learning model used to understand, generate, or parse human language.

[1289] "API" stands for Application Programming Interface, a set of rules and protocols that allow software to communicate with one another.

[1290] A "speech recognition API" is an application programming interface that provides functionality for analyzing voice data and converting it into text.

[1291] "Summary generation" is the process of extracting important information from long text data and summarizing it concisely.

[1292] A "question and answer pair (QA pair)" is a combination of data that associates a suitable answer to a specific question.

[1293] A "database" is a system for systematically storing and managing large amounts of data, and for efficiently accessing, updating, and searching them.

[1294] A "machine learning model" is a collection of algorithms that learn patterns and rules based on large amounts of data and make predictions and classifications.

[1295] "User terminal" refers to a device through which a user accesses information and services, including a PC, smartphone, tablet, etc.

[1296] A "summary" is a concise expression of the main points extracted from detailed information.

[1297] "Searching" is the process of locating specific information in a database or other data source.

[1298] "Topic prediction" is the process of using past data to predict topics that are likely to be covered in future meetings or discussions.

[1299] The present invention is a system for improving the productivity of meetings, and specifically includes the following functions: collecting meeting records, generating summaries using natural language processing, generating question-and-answer pairs, building and searching a database, predicting the next topic of discussion, and providing this information to a user terminal. Below, we will explain the specific procedures for implementing the present invention, as well as the hardware and software used.

[1300] Server Processing

[1301] The server first collects meeting records from the company's conferencing system (e.g., video conferencing, audio conferencing) using an API. For example, the server uses Zoom's API to download the minutes and audio data after the meeting ends. Next, the downloaded audio data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). This text data is then analyzed using a natural language processing model (e.g., the BERT model) to extract the main points of the discussion and generate a summary.

[1302] Based on the generated summaries and meeting records, the server creates question-and-answer pairs (QA pairs). It uses a question generation model to extract questions from text data and uses information retrieval algorithms to obtain appropriate answers. The generated QA pairs are stored in a database.

[1303] Furthermore, the server uses past meeting records to train a machine learning model (e.g., LSTM) to predict the next discussion topic. This model learns past discussion patterns and predicts the topics that are likely to be discussed in the next meeting. The prediction results are also stored in the database.

[1304] Terminal handling

[1305] When a user accesses the meeting tool from their device, the server sends the latest meeting summary to the device and displays it. Specifically, the device sends an API request to the server to retrieve the summary data, and then displays this data to the user.

[1306] When a user inputs a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. The device sends a search query and the best answer is returned and displayed.

[1307] Additionally, the next meeting preparation screen displays the next discussion topic retrieved from the server by the predictive model, allowing users to efficiently prepare for the next meeting.

[1308] User Action

[1309] Users can check the summary before a meeting to quickly understand the content of past meetings. For example, even if they join a meeting late, they can quickly understand the main points discussed by checking the meeting summary.

[1310] During meetings or in preparation, users can use the Q&A search function to instantly obtain the information they need. For example, if they ask, "When is the new product released?", the answer displayed will be "It is scheduled for March 2024."

[1311] Next discussion topics are predicted using machine learning models and displayed to users, allowing them to prepare in advance for topics that are likely to be discussed in the next meeting, such as preparing materials on "online advertising budgets."

[1312] Through these procedures, the present invention can increase the efficiency of meetings and allow users to quickly obtain the information they need.

[1313] Generative AI model and example prompts

[1314] For example, the prompt to the user to get a summary for a particular conference is:

[1315] "View the summary of the weekly meeting for September 30th."

[1316] In this way, the present invention combines various technologies (APIs, speech recognition APIs, natural language processing models, and machine learning models) to automate meeting recording, summary generation, Q&A pair creation, and topic prediction, providing users with an efficient means of obtaining information.

[1317] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1318] Step 1:

[1319] Collection of meeting records

[1320] The server calls the API to retrieve the meeting records. Specifically, the server uses the API key to send a request to the endpoint of the conference system (e.g., a video conference system). The conference system's API returns the text minutes and audio data based on the conference ID. The server stores the returned data in local storage.

[1321] Input: Meeting ID, API key

[1322] Output: Text transcript, audio data

[1323] Step 2:

[1324] Analysis of meeting recordings and transcription of audio data

[1325] The server converts the acquired voice data into text data using a voice recognition API (e.g., Google Cloud Speech-to-Text). Specifically, it sends the voice data to the voice recognition API and saves the returned text data in local storage.

[1326] Input: Audio data

[1327] Output: Text data

[1328] Step 3:

[1329] Meeting Record Analysis and Summary Generation

[1330] The server uses a natural language processing model (e.g., the BERT model) to analyze the text data, extract key arguments and conclusions, and generate a summary. Specifically, the text data is input into the natural language processing model, which extracts the main points of the argument and outputs them as a summary. The server then stores the generated summary in a database.

[1331] Input: Text data

[1332] Output: Summary

[1333] Step 4:

[1334] Building a QA database

[1335] The server generates question and answer pairs (QA pairs) based on the text data and the generated summaries. Specifically, it extracts questions using a question generation model, obtains appropriate answers using an information retrieval algorithm, and stores the QA pairs in a database.

[1336] Input: Text data, summary

[1337] Output: QA pair

[1338] Step 5:

[1339] Prediction of next discussion topic

[1340] The server uses a machine learning model (e.g., LSTM) to learn from past meeting records and predict the next discussion topic. Specifically, it inputs the past meeting record dataset into the model and predicts the next topic to be discussed. The prediction results are stored in a database.

[1341] Input: Past meeting recording data

[1342] Output: Next discussion topic prediction

[1343] Step 6:

[1344] Viewing the Meeting Summary

[1345] When a user accesses the conferencing tool from a device, the latest meeting summary is sent from the server to the device and displayed. Specifically, the device sends an API request to the server, and the server returns the summary data. The device then displays the received summary data to the user.

[1346] Input: API request

[1347] Output: Meeting summary

[1348] Step 7:

[1349] QA search function provided

[1350] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. Specifically, the device sends a search query to the server, which returns the best answer, and the device displays the received answer to the user.

[1351] Input: Question or keyword

[1352] Output: Answers to QA pairs

[1353] Step 8:

[1354] Display next discussion topic

[1355] The next meeting preparation screen displays the next discussion topic obtained from the server to the user. Specifically, the server sends the discussion topic to the terminal based on the prediction model, and the terminal displays this information to the user.

[1356] Input: Predictive model results

[1357] Output: Next discussion topic

[1358] In this way, by combining each processing step, the present invention can efficiently perform processes from collecting meeting records to generating summaries, generating and searching Q&A pairs, and predicting and displaying the next discussion topic.

[1359] (Application example 1)

[1360] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1361] When operating factory robots, it is necessary to quickly identify the cause and take measures when a problem occurs. However, analyzing operation records and audio logs takes time, making it difficult to respond efficiently. In addition, proper preparations are often not made for the next maintenance or adjustment work. This leads to issues such as reduced productivity throughout the factory and increased operating costs.

[1362] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1363] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for collecting operation records in a factory and applying natural language processing to generate analysis and summaries, means for generating and storing question-and-answer pairs from operation data of automated equipment, and means for predicting the topic of the next maintenance or adjustment work. This enables rapid problem solving in the operation of factory robots and efficient preparation for the next work.

[1364] "Meeting records" are text and audio data that record the contents of a meeting.

[1365] A "natural language processing model" is an algorithm or technology that allows computers to understand and analyze human language.

[1366] A summary is a short summary of the main points of a longer piece of text or record.

[1367] A "question and answer pair" is a combination of appropriate answers that correspond to a particular question.

[1368] A "database" is a system for organizing and storing information.

[1369] "Topics to be discussed next" are the main points of discussion at the next meeting predicted based on past meeting records.

[1370] A "user terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[1371] "Factory operation records" are data recorded about the operation of factory robots and equipment.

[1372] "Automation equipment" refers to robots and machines that perform tasks automatically within a factory.

[1373] "Maintenance and adjustment work" refers to the maintenance and correction work carried out to ensure that factory robots and equipment operate normally.

[1374] The present invention relates to a system for improving the operational efficiency of factory robots. Specific embodiments will be described below.

[1375] Server-side processing

[1376] First, as a means of collecting operational records within the factory, each factory robot is equipped with multiple sensors and cameras. This data is sent to a central server. The sensors used include vibration sensors and temperature sensors, and the cameras are optical sensors for video recording.

[1377] The server uses speech recognition software (e.g., the SpeechRecognition library) to convert the audio data into text. After converting the audio file to text, it analyzes it using a natural language processing (NLP) model (e.g., the Transformers library) to generate a summary of the operational recording.

[1378] Next, the server generates question and answer pairs (QA pairs) from the generated summaries and the original operational records. In this process, a question generation algorithm and an answer extraction algorithm are used. The generated QA pairs are stored in a database.

[1379] Furthermore, the server predicts topics that will be required for the next maintenance or adjustment work based on past operation records. Here, machine learning models are trained and predictions are made. The information generated in this way is sent to the factory manager's user terminal and the factory robot itself.

[1380] Terminal side processing

[1381] The user terminal displays a summary of the latest operation records sent from the server, allowing the factory manager to quickly grasp the situation.

[1382] Additionally, when a user enters a specific question or keyword, the device searches the Q&A database on the server and displays related Q&A pairs, allowing users to instantly obtain the information they need during operation or preparation.

[1383] Furthermore, on the screen for preparing for the next maintenance or adjustment work, the user's device displays the next topic obtained from the server's predictive model, allowing the user to efficiently prepare for the next work.

[1384] User processing

[1385] Factory managers can check the summary of operation records and quickly understand the details of past work. This summary function is particularly useful in emergencies and when problems occur, when quick decision-making is required.

[1386] Users can also use the Q&A search function to instantly respond to any questions or problems that arise during operation. For example, if you enter "What is the cause of the robot's picking error?", the result will be "Due to misalignment of the sensor."

[1387] A specific example would be a case where a robot frequently experiences errors during picking operations. When this happens, sensor data and audio logs are recorded and analyzed. At the next maintenance, a predictive model will indicate that "sensor calibration" is required, allowing for advance preparation.

[1388] Prompt Sentence Examples

[1389] "Please summarize the causes and solutions for errors in robot picking operations."

[1390] "Generate QA pairs regarding picking errors."

[1391] In this way, the present invention significantly improves the operational efficiency of factory robots, enabling faster problem solving and more efficient preparation for the next job.

[1392] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1393] Step 1:

[1394] Data collection from factory robots

[1395] The server collects operational data from the sensors and cameras installed on each robot in the factory. Inputs include data from vibration sensors, temperature sensors, and cameras. This data is sent to the server and stored. Data processing involves standardizing the numerical data from each sensor and extracting frames from camera footage. The output is standardized data and video frames that are stored on the server as an operational record.

[1396] Step 2:

[1397] Converting audio data to text

[1398] The server converts the collected voice data into text. This process uses voice recognition software. The input is an audio file, and the output is the operation log converted into text. The audio file is imported, and the voice recognition software analyzes it and transcribes the operation log.

[1399] Step 3:

[1400] Summary generation using natural language processing

[1401] The server analyzes the converted operational records using a natural language processing (NLP) model to generate a summary. The input is the text record, and the output is the summarized text. The NLP model extracts important topics and keywords and generates a shortened summary text based on them.

[1402] Step 4:

[1403] Generate question and answer pairs

[1404] The server generates question and answer pairs (QA pairs) based on the generated summary and the original operation record. The input is the summary text and the operation record, and the output is a list of QA pairs. The QA generation algorithm analyzes the conversation flow and extracts and pairs appropriate questions and answers.

[1405] Step 5:

[1406] QA pair database storage

[1407] The server stores the generated QA pairs in a database. The input is a list of QA pairs and the output is the records stored in the database. A data insertion algorithm adds the QA pairs to the database and generates an index.

[1408] Step 6:

[1409] Prediction of upcoming maintenance topics

[1410] The server uses a machine learning model to predict topics required for the next maintenance or adjustment work from past operation records. The input is past operation records, and the output is a list of next maintenance topics. The machine learning model learns trends from past data and predicts the information required for future maintenance.

[1411] Step 7:

[1412] Summary display on user terminal

[1413] The terminal displays a summary of the latest operational records sent from the server. The input is the summary text sent from the server, and the output is the summary information displayed on the screen of the user terminal. The user interface software receives the summary and displays it in an appropriate format.

[1414] Step 8:

[1415] Execute the QA search function

[1416] When a user inputs a specific question or keyword, the terminal searches the server's QA database and displays related QA pairs. The input is the user's question text, and the output is the answer text retrieved from the QA database. The search algorithm extracts and displays the QA pairs corresponding to the question from the database.

[1417] Step 9:

[1418] Display next maintenance topic

[1419] The terminal displays the topics for the next maintenance or adjustment work obtained from the server's prediction model to the user. The input is a list of predicted maintenance topics, and the output is topic information displayed on the screen of the user's terminal. The user interface receives the topic information and displays it to help prepare for the next time.

[1420] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1421] The present invention is characterized by combining an emotion engine with a system aimed at improving meeting productivity, thereby providing optimal information according to the user's emotions. Specific embodiments of the system will be described below.

[1422] Server-side processing

[1423] 1. Collection of meeting records

[1424] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[1425] Examples:

[1426] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[1427] 2. Meeting Record Analysis and Summary Generation

[1428] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[1429] Examples:

[1430] The audio file is converted into text and key discussion points are extracted, resulting in a summary of the main topics being "budget cuts" and "marketing strategies for new products."

[1431] 3. Emotion evaluation using an emotion engine

[1432] When analyzing the meeting records, the server uses an emotion engine to evaluate the emotions of the participants, allowing it to grasp the emotional tendencies of each speaker and evaluate the flow and atmosphere of the discussion.

[1433] Examples:

[1434] The server evaluates speaker A's voice tone and choice of words during the meeting as "Speaker A has negative emotions" and "Speaker B has positive emotions."

[1435] 4. Building a QA database

[1436] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[1437] Examples:

[1438] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[1439] 5. Predicting the next discussion topic

[1440] The server uses past meeting records, generated summaries, and sentiment data to predict topics that may be discussed in the next meeting. This prediction involves using machine learning models to learn past patterns and infer future topics.

[1441] Examples:

[1442] By learning from past discussion patterns and sentiment data, it is possible to predict that the next meeting will feature a discussion on "increasing online advertising budgets."

[1443] Terminal side processing

[1444] 1. View the meeting summary

[1445] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[1446] Examples:

[1447] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[1448] 2. Displaying information according to user emotions

[1449] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be configured to simplify the display content.

[1450] Examples:

[1451] If the user is rated as "stressed," the summary will be further abridged to show only the main points.

[1452] 3. Q&A search function

[1453] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[1454] Examples:

[1455] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[1456] 4. Display next discussion topic

[1457] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[1458] Examples:

[1459] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[1460] User processing

[1461] 1. Check the summary

[1462] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[1463] Examples:

[1464] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[1465] 2. Using Q&A Search

[1466] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[1467] Examples:

[1468] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[1469] 3. Confirm the next discussion topic

[1470] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[1471] Examples:

[1472] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[1473] The present invention allows each participant to quickly obtain the information they need in real time, and makes it possible to provide optimal information according to the user's emotions, thereby significantly improving the productivity of the conference.

[1474] The processing flow will be explained below.

[1475] Step 1:

[1476] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing systems used by the company (for example, Zoom or Microsoft Teams) using APIs. As a result, the data is accumulated on the server as soon as the meeting ends.

[1477] Step 2:

[1478] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[1479] Step 3:

[1480] The server applies natural language processing (NLP) models to analyze the converted text data. Specifically, it uses IBM Watson's NLP model to analyze the meeting content in order to extract key discussion points and keywords within the text.

[1481] Step 4:

[1482] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main points of discussion and conclusions. For example, a summary such as "budget cuts and new product marketing strategies" is generated.

[1483] Step 5:

[1484] While analyzing the meeting recording, the server uses an emotion engine to assess participants' emotions, specifically, rating speaker A as having a negative emotion and speaker B as having a positive emotion based on their voice tone and word choice.

[1485] Step 6:

[1486] The server generates question and answer pairs (QA pairs) based on the generated summary and the meeting transcript. This process involves using natural language generation (NLG) technology to generate answers to potential questions.

[1487] Step 7:

[1488] The server stores the generated Q&A pairs in a database, which allows the server to quickly provide answers when users ask questions.

[1489] Step 8:

[1490] The server predicts topics that are likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. For prediction, it uses a machine learning model to learn past patterns and infer future topics.

[1491] Step 9:

[1492] When a user receives a notification of an upcoming meeting, the device accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[1493] Step 10:

[1494] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be set to display a simpler version of the information.

[1495] Step 11:

[1496] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays those answers to the user.

[1497] Step 12:

[1498] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[1499] Step 13:

[1500] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[1501] This system allows users to quickly obtain necessary information in real time and provides optimal information according to their emotions, significantly improving the productivity of meetings.

[1502] Example 2

[1503] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1504] Conventional conference systems provide insufficient information to improve conference productivity, making it difficult for users to quickly understand the content of the conference and efficiently prepare for it. Furthermore, it is difficult to provide optimal information according to the emotions of participants in the conference, resulting in a lack of improvement in the overall quality of the conference.

[1505] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records and emotion data, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, and means for displaying information optimal for the user based on emotion data. This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the conference.

[1506] "Meeting records" refer to data that records the contents of a meeting as text, audio data, or the like.

[1507] A "natural language processing model" is a model for processing natural language such as text and speech using a computer, and is used to analyze meeting records.

[1508] A "summary" is a concise summary of the main points and conclusions of a meeting.

[1509] A "question-answer pair" is a set of questions and their corresponding answers extracted from the conference transcripts and summaries.

[1510] A "database" is a structured collection of data that organizes and stores information and makes it searchable and browsable.

[1511] "Emotional data" refers to emotional information analyzed from the statements and attitudes of meeting participants.

[1512] A "machine learning model" is a computer program that learns patterns from data and uses that knowledge to make predictions and classifications.

[1513] A "user terminal" is a device used by a user to access the system, and includes a PC, tablet, smartphone, etc.

[1514] "Optimal information" refers to necessary and appropriate information provided according to the user's situation and emotions.

[1515] "Past meeting records" refers to recorded data from previous meetings, which are used to predict the next discussion topic.

[1516] "Topics to be discussed next" refers to subjects or themes that are likely to be discussed at the next meeting.

[1517] The present invention aims to improve the productivity of meetings by providing optimal information according to the user's emotions using a system that combines an emotion engine. As a specific embodiment of this system, the roles of the server, terminal, and user are described in detail below.

[1518] Server-side processing

[1519] The server collects meeting records, analyzes their contents, generates summaries, and creates question-and-answer pairs (QA pairs) that are stored in a database. It also predicts the next topic to be discussed based on past meeting records and emotion data, and provides optimal information according to the user's emotions.

[1520] Hardware and Software:

[1521] Meeting recording collection: Use Zoom API, Microsoft Teams API, etc.

[1522] Natural Language Processing: Uses the Google Cloud Natural Language API and the BERT model.

[1523] Voice data analysis: Uses Google Cloud Speech-to-Text.

[1524] Sentiment analysis: Uses IBM Watson Tone Analyzer.

[1525] Machine learning model: Uses TensorFlow.

[1526] Examples:

[1527] The server uses the Zoom API to collect meeting audio data and text transcripts, then converts the audio to text using Google Cloud Speech-to-Text. It then analyzes the transcripts using a natural language processing model (e.g., the BERT model) to generate a summary with key discussion points such as "budget cuts" and "new product marketing strategies." It then uses IBM Watson Tone Analyzer to evaluate the speaker's emotions and saves the results in a database. Next, it uses a machine learning model (e.g., TensorFlow) to learn from past meeting recordings and emotion data and predict whether "increasing the online advertising budget" is a topic that may be discussed in the next meeting.

[1528] Terminal side processing

[1529] The terminal provides an interface for users to view the latest meeting summary, emotion evaluation data, Q&A pairs, and upcoming discussion topics provided by the server, and adjusts the displayed information according to the user's emotion.

[1530] Examples:

[1531] When a user logs in to a device, the latest meeting summary is retrieved from the server and displayed. For example, a summary such as "Budget cuts, marketing strategy" may be displayed. If the user is rated as feeling stressed, the display content is further simplified to show only the main points. If a user uses the Q&A search function to ask, "When is the new product being released?", the answer displayed is "It is scheduled for March 2024." As the next meeting date approaches, the device displays an alert such as, "Next meeting topic: Online advertising budget."

[1532] User processing

[1533] Users can view meeting summaries, use the Q&A search function, check upcoming discussion topics, and instantly retrieve important information to respond to emergencies or unexpected situations.

[1534] Examples:

[1535] If a user joins a meeting late, they can check the summary in advance to understand the main points of discussion and quickly get on board with the flow of the meeting. Also, during a meeting, they can use the Q&A search function to search for a question such as "Why are there budget cuts?" and instantly get the answer "Due to rising production costs." By checking the next discussion topic in advance and preparing materials based on it, meetings can proceed smoothly.

[1536] This system allows each participant to quickly obtain the information they need in real time and provides optimal information according to the user's emotions, thereby significantly improving the productivity of meetings.

[1537] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1538] Server-side processing

[1539] Step 1: Collect meeting notes

[1540] Input: Conference system API access key and conference information

[1541] Specific behavior:

[1542] The server accesses the API of the conferencing system used by the company (e.g., Zoom, Microsoft Teams) and authenticates the access key required to obtain the conference list.

[1543] Data processing / calculation: The server makes an API call to obtain a list of target conferences.

[1544] Output: Text transcript of the meeting and audio file

[1545] Step 2: Retrieve meeting data

[1546] Input: Retrieved conference ID and information

[1547] Specific behavior:

[1548] Based on the acquired conference ID, the server uses the API to download the text minutes and audio data of the completed conference.

[1549] Data processing / calculation: The downloaded data is processed into the required format for convenient reference.

[1550] Output: Downloaded text transcript and audio file

[1551] Step 3: Convert audio data to text

[1552] Input: Audio data file

[1553] Specific behavior:

[1554] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[1555] Data processing / calculation: Analyzes language data extracted from speech and converts it into text data.

[1556] Output: Converted text data

[1557] Step 4: Analyzing the text data

[1558] Input: Text data

[1559] Specific behavior:

[1560] The server uses a natural language processing (NLP) model (e.g., the BERT model) to analyze the text data and extract key discussion points and conclusions.

[1561] Data processing / computation: Analyze data using NLP models to extract key information.

[1562] Output: Analysis data with key discussion points and conclusions

[1563] Step 5: Generate a summary

[1564] Input: Extracted discussion points and conclusions

[1565] Specific behavior:

[1566] The server generates a concise meeting summary based on the extracted information.

[1567] Data processing / computation: Applying algorithms to synthesize extracted information and generate summaries.

[1568] Output: Generated meeting summary

[1569] Step 6: Perform sentiment analysis

[1570] Input: Text data

[1571] Specific behavior:

[1572] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze participants' emotions from the text data.

[1573] Data processing / computation: Analyze language data to assess the speaker's sentiment and store it as metadata.

[1574] Output: Emotion rating data

[1575] Step 7: Storing Emotion Data

[1576] Input: Emotion rating data

[1577] Specific behavior:

[1578] The server stores the analyzed emotion data as metadata for the meeting record.

[1579] Data processing / calculation: Formatting the data appropriately for storage and adding it to the database.

[1580] Output: Stored emotion data

[1581] Step 8: Generate question and answer pairs

[1582] Input: Meeting notes, summary data

[1583] Specific behavior:

[1584] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries.

[1585] Data processing / calculation: Extract questions and corresponding answers to form pairs.

[1586] Output: Generated QA pairs

[1587] Step 9: Register with the database

[1588] Input: QA pair

[1589] Specific behavior:

[1590] The server registers the generated QA pairs in the QA database.

[1591] Data processing / calculation: Converting data into the format required for storing it in the database.

[1592] Output: Registered QA pairs

[1593] Step 10: Learn from past data

[1594] Input: Past meeting notes and emotion data

[1595] Specific behavior:

[1596] The server uses machine learning models (e.g., TensorFlow) to train them on past meeting recordings and emotion data.

[1597] Data processing / computation: Using machine learning algorithms to train datasets and build predictive models.

[1598] Output: A trained machine learning model

[1599] Step 11: Future topic prediction

[1600] Input: A trained model

[1601] Specific behavior:

[1602] The server uses the trained model to predict topics that are likely to be discussed in the next meeting.

[1603] Data processing / calculation: Predict future topics based on past data.

[1604] Output: Predicted next discussion topic

[1605] Terminal side processing

[1606] Step 1: Stay up to date

[1607] Input: Request for Updates

[1608] Specific behavior:

[1609] The terminal sends a request to obtain the latest meeting summary from the server.

[1610] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[1611] Output: The latest meeting summary retrieved from the server

[1612] Step 2: View the summary

[1613] Input: Latest meeting summary

[1614] Specific behavior:

[1615] The acquired conference summary is displayed on the terminal.

[1616] Data processing / calculation: Converting data into a concise and easy-to-understand format for display on the UI.

[1617] Output: Displayed meeting summary

[1618] Step 3: Obtaining sentiment rating data

[1619] Input: Emotion rating data request

[1620] Specific behavior:

[1621] The terminal acquires the user's emotion evaluation data from the server.

[1622] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[1623] Output: Emotion rating data obtained from the server

[1624] Step 4: Displaying the best information

[1625] Input: Emotion rating data

[1626] Specific behavior:

[1627] The device is configured to display the most appropriate information for the user based on the emotion data.

[1628] Data processing / calculation: Adjust display priority and content based on emotional data.

[1629] Output: Displaying the most appropriate information according to the user's emotions

[1630] Step 5: Submitting a search query

[1631] Input: Questions or keywords entered by the user

[1632] Specific behavior:

[1633] When a user enters a specific question or keyword, the device sends a request to search the server's Q&A database.

[1634] Data processing / calculation: Sending a search query to the server and receiving the results.

[1635] Output: Search results retrieved from the server

[1636] Step 6: Viewing QA Pairs

[1637] Input: Search results retrieved from the server

[1638] Specific behavior:

[1639] The terminal displays the QA pairs retrieved from the server.

[1640] Data processing / calculation: Converting search results into a format that can be displayed on the UI.

[1641] Output: Display of QA pairs

[1642] Step 7: Get next topic data

[1643] Input: Next discussion topic request

[1644] Specific behavior:

[1645] The terminal obtains the next discussion topic from the server.

[1646] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[1647] Output: Next discussion topic retrieved from the server

[1648] Step 8: View discussion topics

[1649] Input: Next discussion topic data

[1650] Specific behavior:

[1651] The next discussion topic will be displayed on the device.

[1652] Data processing / calculation: Format the display appropriately and present it to the user.

[1653] Output: Show next discussion topic

[1654] User processing

[1655] Step 1: Access the summary

[1656] Input: A request from a user to view a summary

[1657] Specific behavior:

[1658] The user performs an operation on the terminal to check the summary of the conference.

[1659] Data processing / calculation: Taking summary data and converting it into a viewable format.

[1660] Output: A summary of the meeting to be displayed

[1661] Step 2: Enter a search query

[1662] Input: Question or keyword

[1663] Specific behavior:

[1664] The user enters a specific question or keyword into the device's Q&A search field.

[1665] Data processing / calculation: The input data is sent to the server as a search query.

[1666] Output: Trigger to send a search query

[1667] Step 3: Check the search results

[1668] Input: QA pairs obtained from the server

[1669] Specific behavior:

[1670] The user checks the answers displayed as search results.

[1671] Data processing / calculation: Present search results in an easy-to-understand format.

[1672] Output: Display of search results

[1673] Step 4: Confirm the next discussion topic

[1674] Input: Next discussion topic data

[1675] Specific behavior:

[1676] The user checks the next discussion topic and performs operations to prepare in advance.

[1677] Data processing / calculation: Taking topic data and converting it into a viewable format.

[1678] Output: Show next discussion topic

[1679] Step 5: Prepare your materials

[1680] Input: Next discussion topic

[1681] Specific behavior:

[1682] Users prepare the necessary materials based on the next discussion topic.

[1683] Data processing / calculation: Collect and organize materials and compile them into a format that can be used at the next meeting.

[1684] Output: Prepared materials

[1685] This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the meeting.

[1686] (Application example 2)

[1687] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1688] Conventional meeting management systems have difficulty effectively summarizing the contents of meetings and enabling participants to quickly understand the key points. Furthermore, they are unable to provide information that takes into account the emotions of participants, making the outcome of the meeting susceptible to being affected by interpersonal relationships and emotions. Furthermore, they lack the support necessary to efficiently prepare for the next meeting.

[1689] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating a summary, means for analyzing the conference records and evaluating the speaker's emotions, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records, means for displaying the generated summary to a user terminal, means for adjusting the summary content based on the user's emotional state, means for searching for and displaying an answer from a database based on a question from a user terminal, and means for displaying the next topic to be discussed on a user terminal. This allows the content of the conference to be effectively summarized and provided to participants, enabling the provision of information that takes participants' emotions into consideration, and realizing efficient preparation for the next conference.

[1690] "Meeting records" refers to audio data, video data, or text data that records what was said and what was discussed during a meeting.

[1691] "Means of collection" refers to the equipment and software used to obtain meeting records using a conference system or recording device, and send them to a server.

[1692] A "natural language processing model" refers to artificial intelligence technology that analyzes voice and text data and understands their meaning and structure.

[1693] "Summary generation method" refers to algorithms or software that extracts key points and conclusions from meeting records and summarizes them in a concise form.

[1694] "Means for assessing emotions" refers to technology that analyzes speech and tone contained in meeting records to determine the emotional state of the speaker.

[1695] A "question-answer pair" refers to a set of appropriate answers corresponding to a particular question extracted from a meeting transcript or summary.

[1696] A "database" refers to a computer system for systematically storing and managing information such as question and answer pairs.

[1697] "Predictive tools" refers to techniques and algorithms that analyze past data and patterns to predict future events and topics.

[1698] A "machine learning model" refers to an artificial intelligence technology that learns patterns from data and makes predictions and classifications.

[1699] "User terminal" refers to electronic devices used by users, such as personal computers, smartphones, and tablets.

[1700] "Adjusting means" refers to technology or software for varying the amount or format of information displayed based on the user's emotional state.

[1701] "Means for searching and displaying" refers to the technology and software for searching a database based on an inquiry from a user terminal and providing the user with appropriate information.

[1702] "Meeting content" refers to the topics and conclusions discussed at the meeting, as well as the main points made.

[1703] The present invention provides a system for improving productivity of meetings in a factory. Specific embodiments will be described below.

[1704] Server-side processing

[1705] 1. Collection of meeting records

[1706] The server automatically collects meeting records using the factory's conference system and recording devices. The collected audio and video data are sent to the server. As a specific example, the server receives audio files collected from the factory's audio conference system.

[1707] 2. Meeting Record Analysis and Summary Generation

[1708] The server analyzes the collected meeting records using a natural language processing (NLP) model, extracts key discussion points and conclusions, and generates summaries. This process uses the "speech_recognition" library, which converts audio data into text, and the "Hugging Face" library, which performs natural language processing. For example, an audio file can be converted into text and summarized as key topics such as "improving production efficiency" and "marketing strategies for new products."

[1709] 3. Emotion evaluation using an emotion engine

[1710] The server uses an emotion engine to evaluate the speaker's emotion when analyzing the meeting recording. This process uses the "textblob" library to determine the speaker's emotional state. For example, the server may evaluate "Speaker A has a negative emotion" and "Speaker B has a positive emotion."

[1711] 4. Building a QA database

[1712] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries and stores them in a database. This allows for immediate provision of appropriate answers to specific questions. For example, the question "What are some ways to improve production efficiency?" and the answer "Introduce the latest equipment" are stored in the database from the meeting records.

[1713] 5. Predicting the next discussion topic

[1714] The server predicts topics likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. This prediction involves using the machine learning model "scikit-learn" to learn past patterns and infer future topics. For example, based on past discussion patterns and sentiment data, the server predicts that the next meeting will be about "announcement of a new product."

[1715] Terminal side processing

[1716] 1. View the meeting summary

[1717] When a user accesses the system from a terminal, the latest meeting summary is sent from the server to the terminal and displayed. This allows users who join a meeting late to immediately understand the main discussion content. For example, when a user receives a meeting notification, a three-line summary of last week's meeting content ("Improving production efficiency, marketing strategies") is displayed.

[1718] 2. Displaying information according to emotions

[1719] The device displays information that is optimal for the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the display content is adjusted to be simpler. As a specific example, if the user is evaluated as "feeling stressed," the summary content is further simplified and only the main points are displayed.

[1720] 3. Q&A search function

[1721] When a user enters a specific question or keyword, the device searches the server's QA database and displays related QA pairs. This allows users to instantly obtain the information they need during meetings or in preparation. For example, if a user enters "When is the new product released?", the device will display "It is scheduled for March 2024."

[1722] 4. Display next discussion topic

[1723] On the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's predictive model to the user. This allows the user to efficiently prepare for the next meeting. For example, as the next meeting date approaches, the device displays an alert such as "Next meeting topic: New product announcement."

[1724] User processing

[1725] 1. Check the summary

[1726] Users can check the summary before a meeting to quickly understand the content of past meetings. This is particularly useful when joining a meeting late or when prior preparation is insufficient. For example, even if a user joins a meeting late, they can quickly read the meeting summary to confirm the main points of discussion and quickly adapt to the flow of the meeting.

[1727] 2. Using Q&A Search

[1728] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need. This allows for more efficient discussions. For example, if you search for the question "How can we improve production efficiency?" during a meeting, the answer "By introducing the latest equipment" will immediately appear.

[1729] 3. Confirm the next discussion topic

[1730] Users can check the next discussion topic and prepare in advance more efficiently, which helps meetings proceed smoothly. For example, preparing materials for the "new product launch" in advance for the next meeting makes meeting preparations go more smoothly.

[1731] Prompt Sentence Examples

[1732] "Predict what topics will be discussed at the next meeting."

[1733] "Please summarize the meeting transcript below:

[1734] We will discuss ways to improve the factory production line. Resource optimization is necessary to increase production efficiency. At the next meeting, we will discuss the launch of a new product.

[1735] This allows each participant to quickly obtain the information they need in real time, making it possible to provide optimal information according to the user's emotions, significantly improving the productivity of the meeting.

[1736] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1737] Step 1:

[1738] The server collects the meeting records by acquiring audio and video data from the factory's conference system and recording devices, and then transmits the collected data to the server.

[1739] Input: Audio and video data from the factory's conference system and recording equipment

[1740] Data processing: Standardize the data format and transfer it to the server

[1741] Output: Meeting recording data saved on the server

[1742] Step 2:

[1743] The server converts the collected voice data into text using a natural language processing (NLP) model, using the "speech_recognition" library.

[1744] Input: Audio data stored on the server

[1745] Data processing: The process of converting voice data into text.

[1746] Output: Text transcript of the meeting

[1747] Step 3:

[1748] The server analyzes the transcribed meeting notes using a "transformers" (Hugging Face) library, extracts key discussion points and conclusions, and generates a summary.

[1749] Input: Transcribed meeting transcript

[1750] Data Computing: Using natural language processing models to extract key discussion points and generate summaries

[1751] Output: Abridged meeting transcript

[1752] Step 4:

[1753] The server analyzes the transcripts of the meeting using the "textblob" library and evaluates the emotions of each speaker. This evaluation allows the emotional tendencies of each speaker to be understood.

[1754] Input: Transcribed meeting transcript

[1755] Data calculation: Analyze text data using an emotion evaluation model to determine the emotional state

[1756] Output: Emotion rating data

[1757] Step 5:

[1758] The server generates question and answer pairs (QA pairs) from the generated summaries and the converted meeting records, and stores them in a database.

[1759] Input: Translated meeting notes, summaries

[1760] Data computation: The process of generating questions and potential answers

[1761] Output: QA pair data is saved in the database

[1762] Step 6:

[1763] Based on past meeting records and sentiment data, the server uses a machine learning model with "scikit-learn" to predict topics that are likely to be discussed at the next meeting.

[1764] Input: Past meeting records, emotional evaluation data

[1765] Data Computing: Predicting Future Discussion Topics with Machine Learning Models

[1766] Output: Predicted next discussion topic

[1767] Step 7:

[1768] The terminal receives access from the user, acquires the latest conference summary from the server, and displays it on the user terminal.

[1769] Input: Meeting summary data from the server

[1770] Data processing: Displaying summary data in a user-friendly format

[1771] Output: Meeting summary displayed on user's device

[1772] Step 8:

[1773] The terminal adjusts the display content based on the user's emotion evaluation data and provides the user with optimal information.

[1774] Input: Emotional evaluation data, meeting summary

[1775] Data processing: Adjusting the format of information according to the user's emotional state

[1776] Output: The adjusted meeting summary is displayed on the user's device.

[1777] Step 9:

[1778] The terminal receives specific questions or keywords from the user, searches the server's QA database, and displays related QA pairs.

[1779] Input: User questions or keywords

[1780] Data calculation: Database search

[1781] Output: The search results, QA pairs, are displayed on the user's device.

[1782] Step 10:

[1783] The terminal displays the next discussion topic acquired from the server to the user on the next meeting preparation screen.

[1784] Input: Next discussion topic data from the server

[1785] Data processing: Presenting data in a user-friendly format

[1786] Output: Next discussion topic displayed on the user's terminal

[1787] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1788] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1789] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1790] [Fourth embodiment]

[1791] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1792] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1793] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1794] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1795] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1796] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1797] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1798] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1799] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1800] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1801] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1802] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1803] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1804] The present invention is a system aimed at improving meeting productivity, and its main functions include collecting meeting records, generating summaries using natural language processing, generating question and answer pairs, building and searching a database, predicting the next discussion topic, and providing this information to user terminals.

[1805] Server-side processing

[1806] 1. Collection of meeting records

[1807] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[1808] Examples:

[1809] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[1810] 2. Meeting Record Analysis and Summary Generation

[1811] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[1812] Examples:

[1813] The audio file is converted into text, key discussion points are extracted, and a summary is generated with "budget cuts" and "marketing strategies for new products" as the main topics.

[1814] 3. Building a QA database

[1815] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[1816] Examples:

[1817] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[1818] 4. Predicting the next discussion topic

[1819] The server uses past meeting recordings to train a machine learning model to predict topics that are likely to be discussed in the next meeting, allowing users to efficiently prepare for their next meeting.

[1820] Examples:

[1821] By studying past discussion patterns, it predicts that "increasing the online advertising budget" is likely to be discussed at the next meeting.

[1822] Terminal side processing

[1823] 1. View the meeting summary

[1824] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[1825] Examples:

[1826] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[1827] 2. Q&A search function

[1828] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[1829] Examples:

[1830] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[1831] 3. Display next discussion topic

[1832] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[1833] Examples:

[1834] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[1835] User processing

[1836] 1. Check the summary

[1837] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[1838] Examples:

[1839] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[1840] 2. Using Q&A Search

[1841] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[1842] Examples:

[1843] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[1844] 3. Confirm the next discussion topic

[1845] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[1846] Examples:

[1847] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[1848] The present invention allows each participant to quickly obtain necessary information in real time, greatly improving the productivity of the meeting.

[1849] The processing flow will be explained below.

[1850] Step 1:

[1851] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing system (e.g., Zoom or Microsoft Teams). Specifically, it uses an API to retrieve the data as soon as the meeting ends.

[1852] Step 2:

[1853] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[1854] Step 3:

[1855] The server applies natural language processing (NLP) models to analyze the converted text data, which involves extracting key discussion points and keywords within the text. For example, it uses an IBM Watson NLP model to identify key discussion points.

[1856] Step 4:

[1857] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main discussion points and conclusions. For example, "budget cuts and new product marketing strategies" is summarized as the main discussion points.

[1858] Step 5:

[1859] The server generates question and answer pairs (QA pairs) based on the generated summary and meeting transcripts. This process involves using natural language generation (NLG) technology to create appropriate answers to anticipated questions.

[1860] Step 6:

[1861] The server stores the generated Q&A pairs in a database, allowing it to provide quick answers when users ask questions later.

[1862] Step 7:

[1863] The server uses past meeting recordings and generated summaries to predict topics that may be discussed in the next meeting, which involves using machine learning models to learn past patterns and infer future topics.

[1864] Step 8:

[1865] When a user receives a notification of the next meeting, the terminal accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[1866] Step 9:

[1867] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays the answers to the user.

[1868] Step 10:

[1869] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[1870] Step 11:

[1871] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[1872] Example 1

[1873] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1874] In today's business world, improving the productivity of meetings is extremely important. However, manually organizing huge amounts of meeting records, creating summaries, and predicting the next discussion topic is extremely time-consuming and labor-intensive. Another problem is the lack of a way to efficiently review meeting content and quickly retrieve information from QA databases. Therefore, a new system is needed to solve these problems and improve meeting efficiency.

[1875] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1876] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for acquiring meeting records using an API, means for converting voice data to text using a speech recognition API, means for extracting main points of the discussion and generating summaries using a natural language processing model, and means for predicting the next topic to be discussed using a machine learning model. This enables automatic collection of meeting records, generation of summaries, generation and search of Q&A pairs, and prediction of the next topic to be discussed.

[1877] "Meeting records" are data that record what was discussed, what was said, and what decisions were made at a meeting.

[1878] "Natural language processing model" refers to an algorithm or machine learning model used to understand, generate, or parse human language.

[1879] "API" stands for Application Programming Interface, a set of rules and protocols that allow software to communicate with one another.

[1880] A "speech recognition API" is an application programming interface that provides functionality for analyzing voice data and converting it into text.

[1881] "Summary generation" is the process of extracting important information from long text data and summarizing it concisely.

[1882] A "question and answer pair (QA pair)" is a combination of data that associates a suitable answer to a specific question.

[1883] A "database" is a system for systematically storing and managing large amounts of data, and for efficiently accessing, updating, and searching them.

[1884] A "machine learning model" is a collection of algorithms that learn patterns and rules based on large amounts of data and make predictions and classifications.

[1885] "User terminal" refers to a device through which a user accesses information and services, including a PC, smartphone, tablet, etc.

[1886] A "summary" is a concise expression of the main points extracted from detailed information.

[1887] "Searching" is the process of locating specific information in a database or other data source.

[1888] "Topic prediction" is the process of using past data to predict topics that are likely to be covered in future meetings or discussions.

[1889] The present invention is a system for improving the productivity of meetings, and specifically includes the following functions: collecting meeting records, generating summaries using natural language processing, generating question-and-answer pairs, building and searching a database, predicting the next topic of discussion, and providing this information to a user terminal. Below, we will explain the specific procedures for implementing the present invention, as well as the hardware and software used.

[1890] Server Processing

[1891] The server first collects meeting records from the company's conferencing system (e.g., video conferencing, audio conferencing) using an API. For example, the server uses Zoom's API to download the minutes and audio data after the meeting ends. Next, the downloaded audio data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). This text data is then analyzed using a natural language processing model (e.g., the BERT model) to extract the main points of the discussion and generate a summary.

[1892] Based on the generated summaries and meeting records, the server creates question-and-answer pairs (QA pairs). It uses a question generation model to extract questions from text data and uses information retrieval algorithms to obtain appropriate answers. The generated QA pairs are stored in a database.

[1893] Furthermore, the server uses past meeting records to train a machine learning model (e.g., LSTM) to predict the next discussion topic. This model learns past discussion patterns and predicts the topics that are likely to be discussed in the next meeting. The prediction results are also stored in the database.

[1894] Terminal handling

[1895] When a user accesses the meeting tool from their device, the server sends the latest meeting summary to the device and displays it. Specifically, the device sends an API request to the server to retrieve the summary data, and then displays this data to the user.

[1896] When a user inputs a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. The device sends a search query and the best answer is returned and displayed.

[1897] Additionally, the next meeting preparation screen displays the next discussion topic retrieved from the server by the predictive model, allowing users to efficiently prepare for the next meeting.

[1898] User Action

[1899] Users can check the summary before a meeting to quickly understand the content of past meetings. For example, even if they join a meeting late, they can quickly understand the main points discussed by checking the meeting summary.

[1900] During meetings or in preparation, users can use the Q&A search function to instantly obtain the information they need. For example, if they ask, "When is the new product released?", the answer displayed will be "It is scheduled for March 2024."

[1901] Next discussion topics are predicted using machine learning models and displayed to users, allowing them to prepare in advance for topics that are likely to be discussed in the next meeting, such as preparing materials on "online advertising budgets."

[1902] Through these procedures, the present invention can increase the efficiency of meetings and allow users to quickly obtain the information they need.

[1903] Generative AI model and example prompts

[1904] For example, the prompt to the user to get a summary for a particular conference is:

[1905] "View the summary of the weekly meeting for September 30th."

[1906] In this way, the present invention combines various technologies (APIs, speech recognition APIs, natural language processing models, and machine learning models) to automate meeting recording, summary generation, Q&A pair creation, and topic prediction, providing users with an efficient means of obtaining information.

[1907] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1908] Step 1:

[1909] Collection of meeting records

[1910] The server calls the API to retrieve the meeting records. Specifically, the server uses the API key to send a request to the endpoint of the conference system (e.g., a video conference system). The conference system's API returns the text minutes and audio data based on the conference ID. The server stores the returned data in local storage.

[1911] Input: Meeting ID, API key

[1912] Output: Text transcript, audio data

[1913] Step 2:

[1914] Analysis of meeting recordings and transcription of audio data

[1915] The server converts the acquired voice data into text data using a voice recognition API (e.g., Google Cloud Speech-to-Text). Specifically, it sends the voice data to the voice recognition API and saves the returned text data in local storage.

[1916] Input: Audio data

[1917] Output: Text data

[1918] Step 3:

[1919] Meeting Record Analysis and Summary Generation

[1920] The server uses a natural language processing model (e.g., the BERT model) to analyze the text data, extract key arguments and conclusions, and generate a summary. Specifically, the text data is input into the natural language processing model, which extracts the main points of the argument and outputs them as a summary. The server then stores the generated summary in a database.

[1921] Input: Text data

[1922] Output: Summary

[1923] Step 4:

[1924] Building a QA database

[1925] The server generates question and answer pairs (QA pairs) based on the text data and the generated summaries. Specifically, it extracts questions using a question generation model, obtains appropriate answers using an information retrieval algorithm, and stores the QA pairs in a database.

[1926] Input: Text data, summary

[1927] Output: QA pair

[1928] Step 5:

[1929] Prediction of next discussion topic

[1930] The server uses a machine learning model (e.g., LSTM) to learn from past meeting records and predict the next discussion topic. Specifically, it inputs the past meeting record dataset into the model and predicts the next topic to be discussed. The prediction results are stored in a database.

[1931] Input: Past meeting recording data

[1932] Output: Next discussion topic prediction

[1933] Step 6:

[1934] Viewing the Meeting Summary

[1935] When a user accesses the conferencing tool from a device, the latest meeting summary is sent from the server to the device and displayed. Specifically, the device sends an API request to the server, and the server returns the summary data. The device then displays the received summary data to the user.

[1936] Input: API request

[1937] Output: Meeting summary

[1938] Step 7:

[1939] QA search function provided

[1940] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs. Specifically, the device sends a search query to the server, which returns the best answer, and the device displays the received answer to the user.

[1941] Input: Question or keyword

[1942] Output: Answers to QA pairs

[1943] Step 8:

[1944] Display next discussion topic

[1945] The next meeting preparation screen displays the next discussion topic obtained from the server to the user. Specifically, the server sends the discussion topic to the terminal based on the prediction model, and the terminal displays this information to the user.

[1946] Input: Predictive model results

[1947] Output: Next discussion topic

[1948] In this way, by combining each processing step, the present invention can efficiently perform processes from collecting meeting records to generating summaries, generating and searching Q&A pairs, and predicting and displaying the next discussion topic.

[1949] (Application example 1)

[1950] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1951] When operating factory robots, it is necessary to quickly identify the cause and take measures when a problem occurs. However, analyzing operation records and audio logs takes time, making it difficult to respond efficiently. In addition, proper preparations are often not made for the next maintenance or adjustment work. This leads to issues such as reduced productivity throughout the factory and increased operating costs.

[1952] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1953] In this invention, the server includes means for collecting meeting records, means for analyzing the collected meeting records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the meeting records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past meeting records, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, means for collecting operation records in a factory and applying natural language processing to generate analysis and summaries, means for generating and storing question-and-answer pairs from operation data of automated equipment, and means for predicting the topic of the next maintenance or adjustment work. This enables rapid problem solving in the operation of factory robots and efficient preparation for the next work.

[1954] "Meeting records" are text and audio data that record the contents of a meeting.

[1955] A "natural language processing model" is an algorithm or technology that allows computers to understand and analyze human language.

[1956] A summary is a short summary of the main points of a longer piece of text or record.

[1957] A "question and answer pair" is a combination of appropriate answers that correspond to a particular question.

[1958] A "database" is a system for organizing and storing information.

[1959] "Topics to be discussed next" are the main points of discussion at the next meeting predicted based on past meeting records.

[1960] A "user terminal" is a device used by a user, such as a computer, smartphone, or tablet.

[1961] "Factory operation records" are data recorded about the operation of factory robots and equipment.

[1962] "Automation equipment" refers to robots and machines that perform tasks automatically within a factory.

[1963] "Maintenance and adjustment work" refers to the maintenance and correction work carried out to ensure that factory robots and equipment operate normally.

[1964] The present invention relates to a system for improving the operational efficiency of factory robots. Specific embodiments will be described below.

[1965] Server-side processing

[1966] First, as a means of collecting operational records within the factory, each factory robot is equipped with multiple sensors and cameras. This data is sent to a central server. The sensors used include vibration sensors and temperature sensors, and the cameras are optical sensors for video recording.

[1967] The server uses speech recognition software (e.g., the SpeechRecognition library) to convert the audio data into text. After converting the audio file to text, it analyzes it using a natural language processing (NLP) model (e.g., the Transformers library) to generate a summary of the operational recording.

[1968] Next, the server generates question and answer pairs (QA pairs) from the generated summaries and the original operational records. In this process, a question generation algorithm and an answer extraction algorithm are used. The generated QA pairs are stored in a database.

[1969] Furthermore, the server predicts topics that will be required for the next maintenance or adjustment work based on past operation records. Here, machine learning models are trained and predictions are made. The information generated in this way is sent to the factory manager's user terminal and the factory robot itself.

[1970] Terminal side processing

[1971] The user terminal displays a summary of the latest operation records sent from the server, allowing the factory manager to quickly grasp the situation.

[1972] Additionally, when a user enters a specific question or keyword, the device searches the Q&A database on the server and displays related Q&A pairs, allowing users to instantly obtain the information they need during operation or preparation.

[1973] Furthermore, on the screen for preparing for the next maintenance or adjustment work, the user's device displays the next topic obtained from the server's predictive model, allowing the user to efficiently prepare for the next work.

[1974] User processing

[1975] Factory managers can check the summary of operation records and quickly understand the details of past work. This summary function is particularly useful in emergencies and when problems occur, when quick decision-making is required.

[1976] Users can also use the Q&A search function to instantly respond to any questions or problems that arise during operation. For example, if you enter "What is the cause of the robot's picking error?", the result will be "Due to misalignment of the sensor."

[1977] A specific example would be a case where a robot frequently experiences errors during picking operations. When this happens, sensor data and audio logs are recorded and analyzed. At the next maintenance, a predictive model will indicate that "sensor calibration" is required, allowing for advance preparation.

[1978] Prompt Sentence Examples

[1979] "Please summarize the causes and solutions for errors in robot picking operations."

[1980] "Generate QA pairs regarding picking errors."

[1981] In this way, the present invention significantly improves the operational efficiency of factory robots, enabling faster problem solving and more efficient preparation for the next job.

[1982] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1983] Step 1:

[1984] Data collection from factory robots

[1985] The server collects operational data from the sensors and cameras installed on each robot in the factory. Inputs include data from vibration sensors, temperature sensors, and cameras. This data is sent to the server and stored. Data processing involves standardizing the numerical data from each sensor and extracting frames from camera footage. The output is standardized data and video frames that are stored on the server as an operational record.

[1986] Step 2:

[1987] Converting audio data to text

[1988] The server converts the collected voice data into text. This process uses voice recognition software. The input is an audio file, and the output is the operation log converted into text. The audio file is imported, and the voice recognition software analyzes it and transcribes the operation log.

[1989] Step 3:

[1990] Summary generation using natural language processing

[1991] The server analyzes the converted operational records using a natural language processing (NLP) model to generate a summary. The input is the text record, and the output is the summarized text. The NLP model extracts important topics and keywords and generates a shortened summary text based on them.

[1992] Step 4:

[1993] Generate question and answer pairs

[1994] The server generates question and answer pairs (QA pairs) based on the generated summary and the original operation record. The input is the summary text and the operation record, and the output is a list of QA pairs. The QA generation algorithm analyzes the conversation flow and extracts and pairs appropriate questions and answers.

[1995] Step 5:

[1996] QA pair database storage

[1997] The server stores the generated QA pairs in a database. The input is a list of QA pairs and the output is the records stored in the database. A data insertion algorithm adds the QA pairs to the database and generates an index.

[1998] Step 6:

[1999] Prediction of upcoming maintenance topics

[2000] The server uses a machine learning model to predict topics required for the next maintenance or adjustment work from past operation records. The input is past operation records, and the output is a list of next maintenance topics. The machine learning model learns trends from past data and predicts the information required for future maintenance.

[2001] Step 7:

[2002] Summary display on user terminal

[2003] The terminal displays a summary of the latest operational records sent from the server. The input is the summary text sent from the server, and the output is the summary information displayed on the screen of the user terminal. The user interface software receives the summary and displays it in an appropriate format.

[2004] Step 8:

[2005] Execute the QA search function

[2006] When a user inputs a specific question or keyword, the terminal searches the server's QA database and displays related QA pairs. The input is the user's question text, and the output is the answer text retrieved from the QA database. The search algorithm extracts and displays the QA pairs corresponding to the question from the database.

[2007] Step 9:

[2008] Display next maintenance topic

[2009] The terminal displays the topics for the next maintenance or adjustment work obtained from the server's prediction model to the user. The input is a list of predicted maintenance topics, and the output is topic information displayed on the screen of the user's terminal. The user interface receives the topic information and displays it to help prepare for the next time.

[2010] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2011] The present invention is characterized by combining an emotion engine with a system aimed at improving meeting productivity, thereby providing optimal information according to the user's emotions. Specific embodiments of the system will be described below.

[2012] Server-side processing

[2013] 1. Collection of meeting records

[2014] The server automatically collects meeting records from the conference systems (e.g., video conferences, audio conferences) used by the company, using APIs to obtain the necessary data.

[2015] Examples:

[2016] After the meeting ends, the server uses Zoom's API to download the text minutes and audio data for the weekly meeting on September 30th.

[2017] 2. Meeting Record Analysis and Summary Generation

[2018] The server analyzes the acquired meeting records using natural language processing (NLP) models to extract the main points of discussion and conclusions from the meeting and generate a summary, which also includes converting the audio data into text.

[2019] Examples:

[2020] The audio file is converted into text and key discussion points are extracted, resulting in a summary of the main topics being "budget cuts" and "marketing strategies for new products."

[2021] 3. Emotion evaluation using an emotion engine

[2022] When analyzing the meeting records, the server uses an emotion engine to evaluate the emotions of the participants, allowing it to grasp the emotional tendencies of each speaker and evaluate the flow and atmosphere of the discussion.

[2023] Examples:

[2024] The server evaluates speaker A's voice tone and choice of words during the meeting as "Speaker A has negative emotions" and "Speaker B has positive emotions."

[2025] 4. Building a QA database

[2026] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries, and stores them in a database, allowing for immediate provision of appropriate answers to specific questions.

[2027] Examples:

[2028] From the meeting minutes, the question "When is the new product release date?" and the answer "It is scheduled for March 2024" are saved in a database.

[2029] 5. Predicting the next discussion topic

[2030] The server uses past meeting records, generated summaries, and sentiment data to predict topics that may be discussed in the next meeting. This prediction involves using machine learning models to learn past patterns and infer future topics.

[2031] Examples:

[2032] By learning from past discussion patterns and sentiment data, it is possible to predict that the next meeting will feature a discussion on "increasing online advertising budgets."

[2033] Terminal side processing

[2034] 1. View the meeting summary

[2035] When a user accesses the tool from their device, the latest meeting summary is sent from the server to the device and displayed, allowing even users who join the meeting late to immediately understand the main points of discussion.

[2036] Examples:

[2037] When a user receives a meeting notification, a three-line summary of last week's meeting topics ("Budget Cuts, Marketing Strategy") is displayed.

[2038] 2. Displaying information according to user emotions

[2039] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be configured to simplify the display content.

[2040] Examples:

[2041] If the user is rated as "stressed," the summary will be further abridged to show only the main points.

[2042] 3. Q&A search function

[2043] When a user enters a specific question or keyword, the device searches the server's Q&A database and displays related Q&A pairs, allowing users to instantly obtain the information they need during a meeting or in preparation.

[2044] Examples:

[2045] When a user types in "When is the new product released?", the answer displayed is "Scheduled for March 2024."

[2046] 4. Display next discussion topic

[2047] In the next meeting preparation screen, the device displays the next discussion topic retrieved from the server's prediction model to the user, allowing the user to efficiently prepare for the next meeting.

[2048] Examples:

[2049] As the next meeting date approaches, the device will display an alert such as "Next meeting topic: Online advertising budget."

[2050] User processing

[2051] 1. Check the summary

[2052] Users can view the summary before a meeting and quickly understand the content of past meetings, which is especially useful if they arrive late or are not well prepared.

[2053] Examples:

[2054] Even if a user joins a meeting late, they can quickly adapt to the flow of the meeting by reading the meeting summary to identify the main points of discussion.

[2055] 2. Using Q&A Search

[2056] During or in preparation for a meeting, users can use the Q&A search function to instantly obtain the information they need, enabling more efficient discussions.

[2057] Examples:

[2058] During a meeting, you search for the question, "Why are there budget cuts?" and the answer "due to rising production costs" immediately appears.

[2059] 3. Confirm the next discussion topic

[2060] Users can check upcoming discussion topics and prepare in advance more efficiently, which helps meetings run more smoothly.

[2061] Examples:

[2062] Prepare for the next meeting more smoothly by preparing materials related to the "online advertising budget" in advance.

[2063] The present invention allows each participant to quickly obtain the information they need in real time, and makes it possible to provide optimal information according to the user's emotions, thereby significantly improving the productivity of the conference.

[2064] The processing flow will be explained below.

[2065] Step 1:

[2066] The server automatically collects meeting records (minutes, audio data, video data) from the conferencing systems used by the company (for example, Zoom or Microsoft Teams) using APIs. As a result, the data is accumulated on the server as soon as the meeting ends.

[2067] Step 2:

[2068] The server converts the collected voice data into text using voice recognition technology, for example, by using the Google Cloud Speech-to-Text API, which automatically converts voice recordings into text.

[2069] Step 3:

[2070] The server applies natural language processing (NLP) models to analyze the converted text data. Specifically, it uses IBM Watson's NLP model to analyze the meeting content in order to extract key discussion points and keywords within the text.

[2071] Step 4:

[2072] The server generates a summary of the meeting based on the extracted discussion points. The summary includes the main points of discussion and conclusions. For example, a summary such as "budget cuts and new product marketing strategies" is generated.

[2073] Step 5:

[2074] While analyzing the meeting recording, the server uses an emotion engine to assess participants' emotions, specifically, rating speaker A as having a negative emotion and speaker B as having a positive emotion based on their voice tone and word choice.

[2075] Step 6:

[2076] The server generates question and answer pairs (QA pairs) based on the generated summary and the meeting transcript. This process involves using natural language generation (NLG) technology to generate answers to potential questions.

[2077] Step 7:

[2078] The server stores the generated Q&A pairs in a database, which allows the server to quickly provide answers when users ask questions.

[2079] Step 8:

[2080] The server predicts topics that are likely to be discussed in the next meeting based on past meeting records, generated summaries, and sentiment data. For prediction, it uses a machine learning model to learn past patterns and infer future topics.

[2081] Step 9:

[2082] When a user receives a notification of an upcoming meeting, the device accesses the server and displays the latest meeting summary, allowing even users who join late to immediately understand the main discussion content.

[2083] Step 10:

[2084] The device displays the most appropriate information to the user based on the emotion evaluation data obtained from the server. For example, if the user is feeling stressed, the device will be set to display a simpler version of the information.

[2085] Step 11:

[2086] When a user inputs a specific question or keyword into the terminal, the server searches the Q&A database to retrieve related Q&A pairs, and the terminal displays those answers to the user.

[2087] Step 12:

[2088] When a user opens the next meeting preparation screen, the server displays the next discussion topic obtained from the prediction model on the terminal, allowing the user to efficiently prepare for the next meeting.

[2089] Step 13:

[2090] Users can prepare for the next meeting efficiently by collecting necessary materials and information based on predicted discussion topics.

[2091] This system allows users to quickly obtain necessary information in real time and provides optimal information according to their emotions, significantly improving the productivity of meetings.

[2092] Example 2

[2093] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2094] Conventional conference systems provide insufficient information to improve conference productivity, making it difficult for users to quickly understand the content of the conference and efficiently prepare for it. Furthermore, it is difficult to provide optimal information according to the emotions of participants in the conference, resulting in a lack of improvement in the overall quality of the conference.

[2095] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting conference records, means for analyzing the collected conference records using a natural language processing model and generating summaries, means for generating question-and-answer pairs from the conference records and the generated summaries and storing them in a database, means for predicting the next topic to be discussed based on past conference records and emotion data, means for displaying the generated summaries to a user terminal, means for searching for and displaying answers from a database based on questions from the user terminal, means for displaying the next topic to be discussed on the user terminal, and means for displaying information optimal for the user based on emotion data. This allows each participant to quickly obtain the information they need and provides optimal information according to the user's emotions, significantly improving the productivity of the conference.

[2096] "Meeting records" refer to data that records the contents of a meeting as text, audio data, or the like.

[2097] A "natural language processing model" is a model for processing natural language such as text and speech using a computer, and is used to analyze meeting records.

[2098] A "summary" is a concise summary of the main points and conclusions of a meeting.

[2099] A "question-answer pair" is a set of questions and their corresponding answers extracted from the conference transcripts and summaries.

[2100] A "database" is a structured collection of data that organizes and stores information and makes it searchable and browsable.

[2101] "Emotional data" refers to emotional information analyzed from the statements and attitudes of meeting participants.

[2102] A "machine learning model" is a computer program that learns patterns from data and uses that knowledge to make predictions and classifications.

[2103] A "user terminal" is a device used by a user to access the system, and includes a PC, tablet, smartphone, etc.

[2104] "Optimal information" refers to necessary and appropriate information provided according to the user's situation and emotions.

[2105] "Past meeting records" refers to recorded data from previous meetings, which are used to predict the next discussion topic.

[2106] "Topics to be discussed next" refers to subjects or themes that are likely to be discussed at the next meeting.

[2107] The present invention aims to improve the productivity of meetings by providing optimal information according to the user's emotions using a system that combines an emotion engine. As a specific embodiment of this system, the roles of the server, terminal, and user are described in detail below.

[2108] Server-side processing

[2109] The server collects meeting records, analyzes their contents, generates summaries, and creates question-and-answer pairs (QA pairs) that are stored in a database. It also predicts the next topic to be discussed based on past meeting records and emotion data, and provides optimal information according to the user's emotions.

[2110] Hardware and Software:

[2111] Meeting recording collection: Use Zoom API, Microsoft Teams API, etc.

[2112] Natural Language Processing: Uses the Google Cloud Natural Language API and the BERT model.

[2113] Voice data analysis: Uses Google Cloud Speech-to-Text.

[2114] Sentiment analysis: Uses IBM Watson Tone Analyzer.

[2115] Machine learning model: Uses TensorFlow.

[2116] Examples:

[2117] The server uses the Zoom API to collect meeting audio data and text transcripts, then converts the audio to text using Google Cloud Speech-to-Text. It then analyzes the transcripts using a natural language processing model (e.g., the BERT model) to generate a summary with key discussion points such as "budget cuts" and "new product marketing strategies." It then uses IBM Watson Tone Analyzer to evaluate the speaker's emotions and saves the results in a database. Next, it uses a machine learning model (e.g., TensorFlow) to learn from past meeting recordings and emotion data and predict whether "increasing the online advertising budget" is a topic that may be discussed in the next meeting.

[2118] Terminal side processing

[2119] The terminal provides an interface for users to view the latest meeting summary, emotion evaluation data, Q&A pairs, and upcoming discussion topics provided by the server, and adjusts the displayed information according to the user's emotion.

[2120] Examples:

[2121] When a user logs in to a device, the latest meeting summary is retrieved from the server and displayed. For example, a summary such as "Budget cuts, marketing strategy" may be displayed. If the user is rated as feeling stressed, the display content is further simplified to show only the main points. If a user uses the Q&A search function to ask, "When is the new product being released?", the answer displayed is "It is scheduled for March 2024." As the next meeting date approaches, the device displays an alert such as, "Next meeting topic: Online advertising budget."

[2122] User processing

[2123] Users can view meeting summaries, use the Q&A search function, check upcoming discussion topics, and instantly retrieve important information to respond to emergencies or unexpected situations.

[2124] Examples:

[2125] If a user joins a meeting late, they can check the summary in advance to understand the main points of discussion and quickly get on board with the flow of the meeting. Also, during a meeting, they can use the Q&A search function to search for a question such as "Why are there budget cuts?" and instantly get the answer "Due to rising production costs." By checking the next discussion topic in advance and preparing materials based on it, meetings can proceed smoothly.

[2126] This system allows each participant to quickly obtain the information they need in real time and provides optimal information according to the user's emotions, thereby significantly improving the productivity of meetings.

[2127] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2128] Server-side processing

[2129] Step 1: Collect meeting notes

[2130] Input: Conference system API access key and conference information

[2131] Specific behavior:

[2132] The server accesses the API of the conferencing system used by the company (e.g., Zoom, Microsoft Teams) and authenticates the access key required to obtain the conference list.

[2133] Data processing / calculation: The server makes an API call to obtain a list of target conferences.

[2134] Output: Text transcript of the meeting and audio file

[2135] Step 2: Retrieve meeting data

[2136] Input: Retrieved conference ID and information

[2137] Specific behavior:

[2138] Based on the acquired conference ID, the server uses the API to download the text minutes and audio data of the completed conference.

[2139] Data processing / calculation: The downloaded data is processed into the required format for convenient reference.

[2140] Output: Downloaded text transcript and audio file

[2141] Step 3: Convert audio data to text

[2142] Input: Audio data file

[2143] Specific behavior:

[2144] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[2145] Data processing / calculation: Analyzes language data extracted from speech and converts it into text data.

[2146] Output: Converted text data

[2147] Step 4: Analyzing the text data

[2148] Input: Text data

[2149] Specific behavior:

[2150] The server uses a natural language processing (NLP) model (e.g., the BERT model) to analyze the text data and extract key discussion points and conclusions.

[2151] Data processing / computation: Analyze data using NLP models to extract key information.

[2152] Output: Analysis data with key discussion points and conclusions

[2153] Step 5: Generate a summary

[2154] Input: Extracted discussion points and conclusions

[2155] Specific behavior:

[2156] The server generates a concise meeting summary based on the extracted information.

[2157] Data processing / computation: Applying algorithms to synthesize extracted information and generate summaries.

[2158] Output: Generated meeting summary

[2159] Step 6: Perform sentiment analysis

[2160] Input: Text data

[2161] Specific behavior:

[2162] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze participants' emotions from the text data.

[2163] Data processing / computation: Analyze language data to assess the speaker's sentiment and store it as metadata.

[2164] Output: Emotion rating data

[2165] Step 7: Storing Emotion Data

[2166] Input: Emotion rating data

[2167] Specific behavior:

[2168] The server stores the analyzed emotion data as metadata for the meeting record.

[2169] Data processing / calculation: Formatting the data appropriately for storage and adding it to the database.

[2170] Output: Stored emotion data

[2171] Step 8: Generate question and answer pairs

[2172] Input: Meeting notes, summary data

[2173] Specific behavior:

[2174] The server generates question and answer pairs (QA pairs) based on the meeting records and the generated summaries.

[2175] Data processing / calculation: Extract questions and corresponding answers to form pairs.

[2176] Output: Generated QA pairs

[2177] Step 9: Register with the database

[2178] Input: QA pair

[2179] Specific behavior:

[2180] The server registers the generated QA pairs in the QA database.

[2181] Data processing / calculation: Converting data into the format required for storing it in the database.

[2182] Output: Registered QA pairs

[2183] Step 10: Learn from past data

[2184] Input: Past meeting notes and emotion data

[2185] Specific behavior:

[2186] The server uses machine learning models (e.g., TensorFlow) to train them on past meeting recordings and emotion data.

[2187] Data processing / computation: Using machine learning algorithms to train datasets and build predictive models.

[2188] Output: A trained machine learning model

[2189] Step 11: Future topic prediction

[2190] Input: A trained model

[2191] Specific behavior:

[2192] The server uses the trained model to predict topics that are likely to be discussed in the next meeting.

[2193] Data processing / calculation: Predict future topics based on past data.

[2194] Output: Predicted next discussion topic

[2195] Terminal side processing

[2196] Step 1: Stay up to date

[2197] Input: Request for Updates

[2198] Specific behavior:

[2199] The terminal sends a request to obtain the latest meeting summary from the server.

[2200] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[2201] Output: The latest meeting summary retrieved from the server

[2202] Step 2: View the summary

[2203] Input: Latest meeting summary

[2204] Specific behavior:

[2205] The acquired conference summary is displayed on the terminal.

[2206] Data processing / calculation: Converting data into a concise and easy-to-understand format for display on the UI.

[2207] Output: Displayed meeting summary

[2208] Step 3: Obtaining sentiment rating data

[2209] Input: Emotion rating data request

[2210] Specific behavior:

[2211] The terminal acquires the user's emotion evaluation data from the server.

[2212] Data processing / computation: Pulling data through HTTP requests and converting it into a displayable format.

[2213] Output: Emotion rating data obtained from the server

[2214] Step 4: Displaying the best information

[2215] Input: Emotion rating data

[2216] Specific behavior:

[2217] The device is configured to display the most appr...

Claims

1. a means of collecting meeting records; A means for analyzing the collected meeting records using a natural language processing model and generating summaries; a means for generating question and answer pairs from the meeting records and the generated summaries and storing them in a database; A means of predicting the next topic to be discussed based on past meeting records, and means for displaying the generated summary on a user terminal; A means for searching for and displaying an answer from a database based on a question from a user terminal; a means for displaying the next topic to be discussed on a user's device; A system including:

2. The system according to claim 1 , wherein the means for predicting the next topic to be discussed based on past meeting records makes the prediction using a machine learning model.

3. The system of claim 1 , wherein the means for collecting meeting records includes means for converting audio data to text using natural language processing.

Images