system

Abstract

We provide the system. [Solution] A means for acquiring schedule data, A means for searching for relevant information from the performance database and information database based on the acquired schedule data, A means of integrating searched information and automatically generating a document in a specified format, A means of saving the generated document and notifying the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a corporate meeting, a lot of time and labor are required for preparing advance materials, creating meeting minutes after the meeting, and creating answers to questions during the meeting. As a result, employees are forced to work overtime, productivity decreases, and business efficiency deteriorates. The present invention solves such problems and aims to improve the efficiency and shorten the time of tasks related to meetings.

Means for Solving the Problems

[0005] The present invention solves the above problems by the following means. That is,

[0006] means for acquiring schedule data,

[0007] A means for searching for relevant information from the performance database and information database based on the acquired schedule data,

[0008] A means of integrating searched information and automatically generating a document in a specified format,

[0009] The system provides a means of saving the generated documents and notifying the user.

[0010] Furthermore, the present invention provides means for transmitting audio data to a server in real time,

[0011] A means of converting transmitted audio data into text,

[0012] A method for analyzing text data and classifying it by speaker,

[0013] A method for automatically generating meeting minutes based on classified text data,

[0014] The system provides a means of saving the generated meeting minutes and notifying the user.

[0015] Furthermore, the present invention provides means for searching an internal portal and database based on the textual content of a question,

[0016] A means of automatically generating answers based on search results,

[0017] A method for creating an email draft based on the generated response,

[0018] The system provides a means of saving created email drafts and notifying the user.

[0019] This will lead to increased efficiency and reduced time spent on tasks related to meetings.

[0020] "Schedule data" refers to data that records information about meetings and other appointments in a format such as date, time, and location.

[0021] The "Performance Database" is a database that stores data related to past business activities and performance.

[0022] The "Information Database" is a database that stores a company's product information, service information, and other company information.

[0023] The "Means for Automatically Generating Documents" is a system that has the function of collecting multiple data and automatically creating documents based on a specified format.

[0024] "Voice Data" is digital data of voices recorded in meetings or other scenarios.

[0025] The "Means for Converting to Text" is a technology or algorithm for analyzing voice data and converting it into corresponding text.

[0026] The "Means for Analyzing Text Data" is an algorithm or software for analyzing text data, identifying and classifying the speaker, content, etc.

[0027] The "Means for Automatically Generating Meeting Minutes" is a system that has the function of automatically creating meeting minutes summarizing the meeting content based on the analyzed text data.

[0028] The "Intra-company Portal" is a web-based interface for accessing information and databases within a company.

[0029] The "Means for Searching" is a technology or algorithm for searching a database or information system based on a specified query or condition.

[0030] The "Means for Generating Answers" is software or technology for automatically creating answers in natural language based on the acquired information.

[0031] A "method for creating email drafts" refers to a system that automatically creates an email draft, formatting and organizing the content based on responses and other information.

[0032] "Means of notifying the user" refer to algorithms and interfaces used to inform the user about created documents or generated information. [Brief explanation of the drawing]

[0033] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0034] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0035] First, let's explain the terminology used in the following explanation.

[0036] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0037] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0038] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0039] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0040] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0041] [First Embodiment]

[0042] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0043] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0044] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0045] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0046] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0047] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0048] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0049] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0050] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0051] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0052] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0053] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0054] Pre-document creation system

[0055] This invention is a system for improving the efficiency of corporate meetings. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, and efficient processing of questions and answers.

[0056] Implementation Example of Preliminary Document Preparation

[0057] 1. Obtaining schedule data

[0058] The server accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, the server confirms that there is a "sales meeting" scheduled.

[0059] 2. Searching for related data

[0060] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[0061] 3. Automatic document generation

[0062] The server integrates the searched data and automatically creates preliminary materials in specified formats such as PowerPoint and PDF. This allows users to prepare high-quality materials without any extra effort.

[0063] 4. Notifications and saving

[0064] The server saves the completed preliminary documents to the company's document management system and sends a notification to the user. The user checks the notification, downloads the documents, and uses them.

[0065] Automatic meeting minutes creation system

[0066] 1. Collection of audio data

[0067] The terminal records audio data during the meeting in real time and sends it to the server.

[0068] 2. Speech Recognition and Text Conversion

[0069] The server receives the audio data and converts it to text using speech recognition technology. For example, it records the sales manager's statement, "It's time for the sales report," as text.

[0070] 3. Text Classification and Analysis

[0071] The server analyzes the text data and categorizes it by speaker, content, and topic. This automatically organizes the meeting minutes.

[0072] 4. Automatic generation and notification of meeting minutes

[0073] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are documented and saved in the specified format. The user receives a notification that the minutes are complete and reviews their contents.

[0074] Automatic question and answer processing system

[0075] 1. Question collection and text transcription

[0076] Users can type or speak questions during the meeting. The device records the questions and, in the case of spoken questions, converts them to text using speech recognition technology.

[0077] 2. Searching for related information

[0078] The server searches the company portal and databases based on the text-based questions and retrieves relevant information.

[0079] 3. Automatic response generation and email draft creation

[0080] The server automatically generates an answer based on the information it has obtained and creates a draft email. The email draft includes the question and the automatically generated answer.

[0081] 4. Notification and Confirmation

[0082] The server saves the completed email draft and sends a notification to the user. The user receives the notification, reviews the email draft, makes any necessary revisions, and sends it.

[0083] As a concrete example, in sales meetings, automating tasks such as reviewing sales performance, introducing new products, and handling questions and answers during the meeting significantly reduces the user's burden and improves productivity. This allows companies to conduct meetings efficiently and make quick and accurate decisions.

[0084] The following describes the processing flow.

[0085] Processing of the program for creating preliminary documents

[0086] Step 1:

[0087] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[0088] Step 2:

[0089] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[0090] Step 3:

[0091] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[0092] Step 4:

[0093] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[0094] Step 5:

[0095] The server integrates the acquired performance data and company profile data, and automatically generates preliminary materials in a specified format (e.g., PowerPoint).

[0096] Step 6:

[0097] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user.

[0098] Processing of the program for automatically creating meeting minutes

[0099] Step 1:

[0100] The terminal records audio data during the meeting in real time and streams the data to the server.

[0101] Step 2:

[0102] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[0103] Step 3:

[0104] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[0105] Step 4:

[0106] The server automatically generates meeting minutes in a pre-configured format based on the classified text data.

[0107] Step 5:

[0108] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user.

[0109] Program processing for automated question and answer session

[0110] Step 1:

[0111] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[0112] Step 2:

[0113] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[0114] Step 3:

[0115] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[0116] Step 4:

[0117] The server automatically generates an answer based on the search results and creates an email draft. The draft includes the question and the answer.

[0118] Step 5:

[0119] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user reviews the draft and makes revisions as needed.

[0120] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, thereby reducing the burden on employees.

[0121] (Example 1)

[0122] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0123] Traditional meeting management methods presented significant challenges, including the time and effort required for everything from scheduling and preparing materials to taking minutes and handling questions during the meeting. In particular, the inability to automate these processes led to decreased corporate efficiency and a lack of speed in decision-making. Furthermore, the manual collection and organization of information by individual staff members was prone to errors and mistakes.

[0124] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0125] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for searching the company portal and database based on the transcribed question content, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This enables the automation and efficiency of the entire meeting management process.

[0126] "Schedule data" refers to information including the scheduled dates and times of meetings and tasks, and is used as the basis for various management and coordination activities.

[0127] A "performance database" is a database that stores data on actual achievements, such as past performance and activity results.

[0128] An "information database" is a database used to store and manage various types of information necessary for business operations, such as product information and customer information.

[0129] "Methods for automatically generating documents" refer to technologies and methods that automatically create documents by integrating related data in a specified format (e.g., PDF or PowerPoint).

[0130] "Means of notifying users" refers to technologies and methods for informing users about generated documents and related information through email or in-application notifications.

[0131] "Audio data" refers to the data format of audio information recorded during meetings or other similar events.

[0132] "Means of transmitting audio data to a server in real time" refers to technologies and methods for sending audio collected during meetings, etc., to a server in real time.

[0133] "Means for converting audio data into text" refers to speech recognition technologies and methods for converting recorded audio into textual information.

[0134] "Text data" refers to character information converted from speech, as well as other text-based data.

[0135] "Methods for analyzing text data and classifying it by speaker" refers to technologies and methods that analyze text data converted from speech and classify and organize information for each speaker.

[0136] "Methods for automatically generating meeting minutes" refers to technologies and methods for automatically creating meeting minutes based on collected and organized text data.

[0137] "Question content" refers to information including questions and concerns raised during the meeting.

[0138] An "internal company portal" is a portal site or system that provides access to information and resources shared within a company.

[0139] "Means of generating answers" refers to technologies and methods that automatically create appropriate answers based on the content of a question.

[0140] A "draft email" is a draft of an email that has been prepared before sending, containing the necessary content but requiring final review and revisions.

[0141] Modes for carrying out the invention

[0142] This invention is a system for improving the efficiency of corporate meetings, providing automatic creation of pre-meeting materials, automatic creation of meeting minutes, and streamlined question-and-answer processing. This system consists of a server, terminals, and users, each performing a specific function.

[0143] Implementation Example of Preliminary Document Preparation

[0144] The server first accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, it might use the "Google® Calendar API." Through this API, it retrieves the schedule data in JSON format.

[0145] Next, the server uses the acquired schedule data to search for relevant information from the company's performance database and information database. This allows it to retrieve past sales performance and product information. For example, it can use SQL Server or MongoDB to retrieve the necessary data from each database via queries.

[0146] To integrate the searched data and automatically create preliminary materials in specified formats such as PowerPoint and PDF, the server uses libraries such as "python-pptx" and "ReportLab". This allows for the automatic generation of materials by displaying performance data in graphs and placing product information in tables and text boxes.

[0147] The completed documents are saved to the document management system using the SharePoint API. The server then sends a notification to the user. Notification methods include email using an SMTP server and in-app notifications via Microsoft Teams.

[0148] Automatic meeting minutes creation system

[0149] The terminal (a device with a microphone in the conference room) records audio data in real time during the meeting and sends it to the server. The audio data is captured via a "sound card" and streamed to the server using WebSocket.

[0150] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text. This text data is returned in JSON format.

[0151] Next, the server uses a natural language processing library such as "SpaCy" to analyze the text data and classify it by speaker. After classification, it automatically generates meeting minutes based on the classified text data.

[0152] The server uses the Google Docs API to document the meeting minutes and saves the generated minutes to Google Drive. The user is then notified of the URL of the meeting minutes.

[0153] Automatic question and answer processing system

[0154] Users can type questions via their device or speak them aloud during the meeting. If they speak, the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[0155] Based on the transcribed query, the server uses "ElasticSearch®" to search for relevant information from the company's internal portal and databases.

[0156] Based on the acquired information, the server automatically generates a response using a generative AI model (e.g., "GPT-3®") and creates an email draft using the "Gmail API". The email draft is saved as a draft.

[0157] Finally, a notification is sent to the user, who can review the draft, make any necessary revisions, and send the email.

[0158] Specific example

[0159] For example, in a sales meeting, by inputting the following prompt into the AI ​​model, it is possible to understand the specific steps for creating preliminary materials.

[0160] Example of a prompt:

[0161] For next week's sales meeting, please automatically generate a PowerPoint presentation containing sales performance data for the past six months and information on new products.

[0162] This prompt indicates how the process will proceed. The server will use the Google Calendar API to retrieve the schedule, search for data in SQL Server and MongoDB, and generate a document using python-pptx. Finally, it will use the SharePoint API to save the document and notify the user.

[0163] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0164] System program processing flow

[0165] Implementation Example of Preliminary Document Preparation

[0166] Step 1:

[0167] The server uses the Google Calendar API to retrieve the company's meeting schedule for the following day. This involves the server sending a request to the API and receiving schedule data in JSON format as a response. For example, it might confirm that a "sales meeting" is scheduled for 2:00 PM the following day.

[0168] Input: Request to the Google Calendar API

[0169] Output: Schedule data in JSON format

[0170] Specific operation: Parse JSON data and extract meeting type, date and time, and participant information.

[0171] Step 2:

[0172] Based on the acquired schedule data, the server searches for relevant data such as past sales performance and product information from SQL Server and MongoDB. This allows the server to execute SQL queries to retrieve the necessary performance data and MongoDB queries to retrieve product information.

[0173] Input: Schedule data

[0174] Output: Search results from the performance database and information database.

[0175] Specific operation: Issue specific queries to SQL Server and MongoDB and collect relevant data.

[0176] Step 3:

[0177] The server uses libraries such as "python-pptx" and "ReportLab" to automatically generate preliminary materials in PowerPoint and PDF formats from the integrated data. The server displays past sales performance as graphs and places new product information as tables and text on the slides.

[0178] Input: Performance data and product information

[0179] Output: Preliminary materials in PowerPoint or PDF format

[0180] Specific actions: Format the data and generate slides according to the specified template.

[0181] Step 4:

[0182] The server saves the generated documents to the company's document management system using the SharePoint API. The server then sends a completion notification to the user.

[0183] Input: Generated preliminary data

[0184] Output: URL of documents stored in SharePoint and user notification

[0185] Specific actions: Upload documents to SharePoint and send email notifications via an SMTP server.

[0186] Automatic meeting minutes creation system

[0187] Step 1:

[0188] The terminal records audio data during the meeting in real time and sends it to the server. The terminal captures the audio via a "sound card" and streams the audio data to the server using WebSocket.

[0189] Input: Real-time audio data

[0190] Output: Audio data sent to the server

[0191] Specific operation: Captures audio input from the microphone and transfers the data via a WebSocket connection.

[0192] Step 2:

[0193] The server uses the Google Cloud Speech-to-Text API to convert received audio data into text. It sends audio data to the API and receives text data as a response.

[0194] Input: Audio data

[0195] Output: Text data

[0196] Specific operation: Divide the audio data into fixed batches, send them to the API, and retrieve the conversion results.

[0197] Step 3:

[0198] The server uses "SpaCy" to analyze text data and classify it by speaker. It extracts speaker names and topics from the text data and categorizes them accordingly.

[0199] Input: Text data

[0200] Output: Classified text data

[0201] Specific operation: Performs text analysis, extracts and classifies speaker names and keywords.

[0202] Step 4:

[0203] The server automatically generates meeting minutes based on text data categorized using the Google Docs API. The generated meeting minutes are saved to Google Drive.

[0204] Input: Classified text data

[0205] Output: Meeting minutes saved to Google Drive

[0206] Specific actions: Insert data into a template, generate a document, and save it.

[0207] Step 5:

[0208] The server will notify the user of the URL of the generated meeting minutes. Notification methods include email and in-app notifications.

[0209] Input: URL of the generated meeting minutes

[0210] Output: Completion notification to the user

[0211] Specific operation: Use an SMTP server to send email notifications and generate in-app notifications.

[0212] Automatic question and answer processing system

[0213] Step 1:

[0214] Users can type or speak questions during the meeting. The device records the questions and, if spoken, converts them to text using the Google Cloud Speech-to-Text API.

[0215] Input: Question in voice or text format

[0216] Output: Text version of the question content

[0217] Specific operation: The user either types their speech or captures the audio and converts it to text.

[0218] Step 2:

[0219] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[0220] Input: Text-based question content

[0221] Output: Search Results

[0222] Specific operation: Send a query to Elasticsearch and retrieve relevant information.

[0223] Step 3:

[0224] Based on the acquired information, the server automatically generates answers using a generative AI model (e.g., "GPT-3").

[0225] Input: Search Results

[0226] Output: Auto-generated answer

[0227] Specific operation: Input prompts into the generative AI model and generate responses.

[0228] Step 4:

[0229] The server uses the Gmail API to generate the response, then creates and saves an email draft.

[0230] Input: Auto-generated answer

[0231] Output: Email draft

[0232] Specific operation: Create and save an email draft using the Gmail API.

[0233] Step 5:

[0234] The user is notified of the completed email draft. The user reviews the draft, makes any necessary revisions, and sends the email.

[0235] Input: Email draft

[0236] Output: Completion notification and correction / sent email to the user.

[0237] Specific operation: An SMTP server is used to send a notification, and the user sends an email after making corrections.

[0238] (Application Example 1)

[0239] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0240] In modern factories, meetings and work reports cover a wide range of topics, requiring significant time and effort for preparation and execution. In particular, creating preliminary materials, preparing meeting minutes, and responding quickly to questions are laborious tasks. This situation leads to decreased meeting efficiency and hinders productivity improvements. A system is needed to address these challenges and streamline meetings and work reports within factories.

[0241] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0242] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a performance database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for recording audio data during a meeting in real time and sending it to the server, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for converting the content of questions entered during the meeting into text and searching the company portal and database, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, means for saving the created email draft and notifying the user, and means for generating answers based on prompt sentences using a generation AI model. As a result, the creation of pre-meeting materials, the creation of meeting minutes, and the processing of questions and answers are automated, enabling efficient meeting progress and rapid decision-making.

[0243] "Schedule data" refers to data that shows the planned dates for meetings and tasks.

[0244] A "performance database" is a database that stores records of past work and meetings.

[0245] An "information database" is a database that stores various kinds of related information.

[0246] "Specified format" refers to a standard that specifically instructs the format and layout of a document.

[0247] A "document" refers to a digital document or paper-based material that organizes information according to a specific format.

[0248] "Audio data" refers to data that includes audio waveform data and its digitized form.

[0249] "Text data" refers to data obtained by converting audio data into written text.

[0250] "Speakers" refer to individuals or groups who make statements during meetings or discussions.

[0251] "Meeting minutes" are documents that record the content of meetings or discussions.

[0252] "Questions" refers to the content of questions submitted during meetings or discussions.

[0253] An "internal portal" is an information sharing platform used within a company.

[0254] A "database" is a system that systematically stores and manages data in a searchable format.

[0255] A "draft email" is a draft of an email document before it is sent.

[0256] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to perform data processing or document generation.

[0257] A "prompt sentence" is a sentence containing instructions or questions that are input into a generative AI model.

[0258] This invention is a system aimed at improving the efficiency of meetings and work reports within a factory, and a specific embodiment thereof is shown below. This system includes a server, terminals, and user operation.

[0259] server

[0260] The server uses the following hardware and software:

[0261] Hardware: High-performance processor, ample memory, SSD storage

[0262] Software: Schedule management system, performance database, information database, natural language processing engine, speech recognition engine

[0263] The server first accesses the company's schedule management system and retrieves schedule data. Based on the retrieved schedule data, it searches for relevant information from the performance database and information database, integrates this information, and automatically generates preliminary materials in the specified format (e.g., PDF or PowerPoint). These generated materials are stored in the company's document management system and notified to the user.

[0264] Furthermore, the server receives audio data transmitted from terminals during the meeting in real time and converts it into text using a speech recognition engine. The converted text data is analyzed by a natural language processing engine and classified by speaker. Based on the classified text data, meeting minutes are automatically generated, saved, and then notified to the user.

[0265] Furthermore, the system transcribes questions entered during meetings into text, searches the company portal and databases to retrieve relevant information, and automatically generates answers based on that information, creating a draft email. This draft email is saved and the user is notified. The system also has a function where the generation AI model generates answers based on prompt text.

[0266] terminal

[0267] The devices are smartphones, tablets, or factory robots used within the factory. They are used as follows:

[0268] Collection of audio data

[0269] Record of questions entered during the meeting

[0270] Receipt and confirmation of notification

[0271] The terminal records audio data from the meeting in real time and sends it to the server. Users can also use the terminal to input questions during the meeting. The terminal receives notifications and informs users when materials or meeting minutes are complete.

[0272] User

[0273] The user performs the following actions:

[0274] Receive notifications from the system

[0275] Review the prepared preliminary documents and make any necessary corrections.

[0276] Review the automatically generated meeting minutes and make corrections if necessary.

[0277] Review the draft email, make any necessary corrections, and then send it.

[0278] Specific example

[0279] As a specific example, the following prompt sentences are used as input to the generative AI model:

[0280] Prompt: Based on the schedule data and past performance data, please create the meeting materials for the next day.

[0281] Data:

[0282] Schedule: Schedule for the next day: Safety meeting

[0283] Performance: Past safety reports

[0284] In this way, the creation of advance materials, the creation of meeting minutes, and the handling of Q&A are automated, significantly improving the efficiency of meetings and work reports within the factory.

[0285] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0286] Step 1:

[0287] The server accesses the company's schedule management system to obtain schedule data. The obtained schedule data serves as the input for this processing. Based on this data, the server proceeds to the next processing step.

[0288] Step 2:

[0289] The server searches for relevant information from the performance database and the information database based on the obtained schedule data. As a result of the search, performance data and relevant information are output. This output is used for the creation of advance materials.

[0290] Step 3:

[0291] The server integrates the performance data and relevant information obtained in Step 2 and automatically generates a document in the specified format. Specifically, it is output as a file in PDF or PowerPoint format. This output document is used as advance materials.

[0292] Step 4:

[0293] The server saves the generated document to the company's document management system and sends a notification to the user when saving is complete. The user receives the notification and downloads and reviews the document as needed.

[0294] Step 5:

[0295] The terminal records audio data during the meeting in real time and sends that data to the server. The audio data arrives at the server as input and proceeds to the next processing step.

[0296] Step 6:

[0297] The server converts the transmitted audio data into text data using a speech recognition engine. This converted text data is then output and analyzed.

[0298] Step 7:

[0299] The server analyzes the converted text data using a natural language processing engine and classifies it by speaker. Text data is used as input, and the classified text data is obtained as output.

[0300] Step 8:

[0301] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are output in a specific format and stored in the company's document management system.

[0302] Step 9:

[0303] The server sends a notification to the user when the generated meeting minutes have been saved. The user receives the notification and reviews the minutes, making any necessary corrections.

[0304] Step 10:

[0305] The text of the question entered by the user using the terminal during the meeting is converted into text, and this text data is sent to the server. The server analyzes this text data and searches the company portal and database to obtain relevant information.

[0306] Step 11:

[0307] Based on the obtained relevant information, the server automatically generates an answer using the generated AI model. The prompt text and relevant information are used as input, and text data as the answer is output.

[0308] Step 12:

[0309] Based on the generated answer, the server creates a draft of the email, saves this draft, and notifies the user. The user receives the notification, checks the content of the email if necessary, makes corrections, and then sends it.

[0310] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0311] Pre - meeting material creation system

[0312] The present invention is a system for improving the efficiency of corporate meetings, and by further considering the user's emotion, it is characterized by generating more personalized materials, meeting minutes, and question - and - answer responses. This system provides automatic creation of pre - meeting materials, automatic creation of meeting minutes, efficiency improvement of question - and - answer processing, and adjustment by an emotion engine.

[0313] Embodiment of pre - meeting material creation

[0314] 1. Acquisition of schedule data

[0315] The server accesses the company's schedule management system to check the meeting schedule for the next day. For example, it might find that there is a meeting named "Sales Meeting".

[0316] 2. Searching for related data

[0317] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[0318] 3. Emotion recognition by an emotion engine

[0319] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[0320] 4. Automatic document generation and adjustment

[0321] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[0322] 5. Notifications and saving

[0323] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[0324] Automatic meeting minutes creation system

[0325] 1. Collection of audio data

[0326] The terminal records audio data during the meeting in real time and streams it to the server.

[0327] 2. Speech Recognition and Text Conversion

[0328] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[0329] 3. Text Classification and Analysis

[0330] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[0331] 4. Analysis using an emotional engine

[0332] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[0333] 5. Automatic generation and adjustment of meeting minutes

[0334] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[0335] 6. Saving and Notifications

[0336] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[0337] Automatic question and answer processing system

[0338] 1. Question collection and text transcription

[0339] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[0340] 2. Text conversion using speech recognition

[0341] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[0342] 3. Searching for related information

[0343] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[0344] 4. Feedback from the Emotion Engine

[0345] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[0346] 5. Automatic response generation and email draft creation

[0347] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[0348] 6. Notification and Confirmation

[0349] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[0350] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, and provide advanced personalization that takes user emotions into account, thereby reducing the burden on employees and improving productivity.

[0351] The following describes the processing flow.

[0352] Processing of the program for creating preliminary documents

[0353] Step 1:

[0354] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[0355] Step 2:

[0356] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[0357] Step 3:

[0358] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[0359] Step 4:

[0360] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[0361] Step 5:

[0362] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[0363] Step 6:

[0364] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[0365] Step 7:

[0366] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[0367] Processing of the program for automatically creating meeting minutes

[0368] Step 1:

[0369] The terminal records audio data during the meeting in real time and streams the data to the server.

[0370] Step 2:

[0371] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[0372] Step 3:

[0373] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[0374] Step 4:

[0375] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[0376] Step 5:

[0377] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[0378] Step 6:

[0379] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[0380] Program processing for automated question and answer session

[0381] Step 1:

[0382] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[0383] Step 2:

[0384] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[0385] Step 3:

[0386] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[0387] Step 4:

[0388] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[0389] Step 5:

[0390] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[0391] Step 6:

[0392] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[0393] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, provide advanced personalization that takes user emotions into account, reduce employee workload, and improve productivity.

[0394] (Example 2)

[0395] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0396] Traditional meeting management systems often require manual processes for meeting preparation, minute-taking, and Q&A, resulting in significant time and effort. Furthermore, these tasks often fail to consider user emotions, impacting meeting efficiency and participant satisfaction. Therefore, there is a need for a system that automates meeting preparation, recording, and Q&A, while also providing personalized responses that take user emotions into account.

[0397] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0398] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a history database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for adjusting the representation of the data using an emotion analysis engine when generating the document, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for analyzing emotions based on the text data, means for automatically generating meeting minutes based on the classified and emotion-analyzed text data, means for searching an internal portal and a database based on the transcribed question content, means for automatically generating answers based on the search results, means for adjusting the representation of the answers using an emotion analysis engine when generating the answers, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This automates meeting preparation, recording, and question handling, and enables advanced personalization that takes user emotions into consideration.

[0399] "Schedule data" refers to data that includes a user's schedule and appointments. It is obtained from the company's schedule management system.

[0400] A "history database" refers to a database that stores data on past achievements and activities.

[0401] An "information database" refers to a database that stores data such as product information and company policies.

[0402] An "emotion analysis engine" refers to software or a system that analyzes a user's emotions from text or audio and outputs the results as emotion data.

[0403] "Documents" refer to digital files containing organized and edited information such as meeting materials, minutes, and email drafts.

[0404] "Specified format" refers to the predetermined format and layout used when creating a document.

[0405] "Audio data" refers to data that digitally saves the content of participants' remarks recorded during a meeting.

[0406] "Text" refers to data obtained by converting audio data into written information.

[0407] "Classification" refers to dividing speakers and their statements into specific categories based on text data.

[0408] "Meeting minutes" refers to a document that records the content and statements made during a meeting.

[0409] An "internal portal" refers to a web-based platform used for information sharing and communication within a company.

[0410] "Answer" refers to the information provided as a response to a question.

[0411] A "draft email" refers to the content of an email that has been saved in its state before being sent to the user.

[0412] This invention relates to a system that automates meeting preparation, minute-taking, and question-and-answer processing, and further generates more personalized materials, minutes, and Q&A by taking user emotions into consideration. This system mainly consists of a server, terminals, and users, and is implemented using the following specific hardware and software.

[0413] Pre-document creation system

[0414] 1. Hardware and software configuration

[0415] The server accesses a schedule management system (e.g., Google Calendar) to retrieve data. Python is the primary programming language, and MySQL® and PostgreSQL are used as databases.

[0416] The server uses an emotion analysis engine (e.g., IBM Watson®) to analyze the user's emotions and use the information to adjust the content of the materials.

[0417] The server uses a combination of Python libraries to automatically generate pre-recorded materials in PowerPoint or PDF format (e.g., python-pptx, ReportLab).

[0418] 2. Specific Examples

[0419] The server retrieves the schedule data for the next day's "sales meeting" and searches the database for past sales performance data and the latest product information. Using an emotion analysis engine, it automatically generates pre-meeting materials that heavily utilize positive language, based on data showing that the user exhibited high stress levels in past sales meetings.

[0420] Example prompts for generative AI models

[0421] Please prepare the preliminary materials for the next day's "Sales Meeting" using the following information and maintaining a positive tone.

[0422] September sales: ¥10,000,000

[0423] New product XYZ specification update

[0424] To reduce the user's past stress

[0425] Automatic meeting minutes creation system

[0426] 1. Hardware and software configuration

[0427] The terminal uses its microphone to record audio data during the meeting and utilizes WebSocket to stream it to the server in real time.

[0428] The server uses the Google Speech-to-Text API to convert the audio data into text.

[0429] The server uses natural language processing tools (e.g., NLTK) to analyze text data and classify it by speaker. It also uses a sentiment analysis engine to analyze emotions.

[0430] 2. Specific Examples

[0431] The terminal records the audio during the meeting and sends it to the server in real time. The server converts the audio, such as "It's time for the sales report," into text, classifies the speaker and the content of their statement, and also performs sentiment analysis.

[0432] Example prompts for generative AI models

[0433] Please create meeting minutes that include the following audio data. Please pay attention to the emotions of the speakers.

[0434] Sales Manager: "It's time for the sales report."

[0435] Emotional data: "Anxiety"

[0436] Automatic question and answer processing system

[0437] 1. Hardware and software configuration

[0438] Users can type questions into their devices or speak them aloud during the meeting. The devices record these questions and send them to the server.

[0439] The server uses the Google Speech-to-Text API to convert voice questions into text and then uses SQL queries to search for relevant information in the company's database.

[0440] The server uses an emotion analysis engine to analyze the questioner's emotions and adjusts the tone of the response accordingly.

[0441] 2. Specific Examples

[0442] If a user asks "What are next month's sales targets?" during a meeting, the device records the question and sends it to the server. The server converts the audio to text, searches for relevant information, and generates a polite response based on sentiment analysis.

[0443] Example prompts for generative AI models

[0444] Please generate answers to the following questions in a polite tone.

[0445] Question text: "What are your sales targets for next month?"

[0446] Emotional data: "Dissatisfaction"

[0447] Through the system configuration and processing steps described above, this invention automates meeting preparation, recording, and question handling, while also providing a high level of personalization that takes user emotions into consideration. This is expected to reduce the burden on employees and improve productivity.

[0448] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0449] Pre-document creation system

[0450] Step 1:

[0451] The server sends an HTTP request to the company's scheduling management system (e.g., Google Calendar) and retrieves the next day's meeting schedule data in JSON format.

[0452] input:

[0453] API endpoint for the schedule management system

[0454] HTTP Request

[0455] output:

[0456] Retrieved schedule data (JSON format)

[0457] Specific actions:

[0458] The server sends an HTTP request and receives schedule data in JSON format. For example, it retrieves data for "October 11, 2023, 9:00 - 10:00 Sales Meeting".

[0459] Step 2:

[0460] Based on the retrieved schedule data, the server issues SQL queries to search for the necessary data from the history database (e.g., MySQL) and the information database (e.g., PostgreSQL).

[0461] input:

[0462] Acquired schedule data

[0463] SQL query

[0464] output:

[0465] Search results include historical data and product information.

[0466] Specific actions:

[0467] Based on the acquired schedule data, the server issues an SQL query to retrieve historical data such as "Sales in September 2023: ¥10,000,000" and product information such as "Specification update for new product XYZ".

[0468] Step 3:

[0469] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the emotion data the user expressed in previous meetings.

[0470] input:

[0471] Past meeting data (audio or text)

[0472] output:

[0473] Emotional data (categories and scores such as stress, satisfaction, and anxiety)

[0474] Specific actions:

[0475] The server feeds audio data from past sales meetings into an emotion analysis engine and obtains analysis results indicating that the user showed high stress levels during past meetings.

[0476] Step 4:

[0477] The server integrates the acquired historical data and product information, and uses a Python script to automatically generate preliminary materials in specified formats such as PowerPoint and PDF.

[0478] input:

[0479] Historical data

[0480] Product Information

[0481] Emotional data

[0482] output:

[0483] Automatically generated pre-document materials (PowerPoint or PDF format)

[0484] Specific actions:

[0485] The server integrates historical data and product information, and creates preliminary materials in PowerPoint format, adjusting the tone of the materials based on sentiment data. For example, it reduces user stress by adding positive language.

[0486] Step 5:

[0487] The server uploads the generated documents to the company's cloud storage (e.g., Google Drive) and sends a completion notification to the user via an HTTP POST request.

[0488] input:

[0489] Automatically generated preliminary materials

[0490] Cloud storage API endpoints

[0491] output:

[0492] Upload to cloud storage complete.

[0493] Notification to the user

[0494] Specific actions:

[0495] The server uploads the created preliminary documents to Google Drive and sends a completion notification to the user via email. The user receives the email, downloads the documents, and reviews them.

[0496] Automatic meeting minutes creation system

[0497] Step 1:

[0498] The terminal uses its microphone to record audio data during the meeting and streams it to the server in real time via WebSocket.

[0499] input:

[0500] Meeting audio

[0501] output:

[0502] Streamed audio data

[0503] Specific actions:

[0504] The device picks up all audio during the meeting using its microphone and transmits it to the server in real time.

[0505] Step 2:

[0506] The server sends the received audio data to the Google Speech-to-Text API to convert the audio into text.

[0507] input:

[0508] Audio data

[0509] output:

[0510] Text data

[0511] Specific actions:

[0512] The server converts the audio "It's time for the sales report" into text "It's time for the sales report".

[0513] Step 3:

[0514] The server analyzes the acquired text and classifies the data based on factors such as the speaker's name, the content of the statement, and the topic. It uses natural language processing tools (e.g., NLTK).

[0515] input:

[0516] Text data

[0517] output:

[0518] Classified text data

[0519] Specific actions:

[0520] The server analyzes the text data and categorizes it as "Sales Manager" or "Sales Report."

[0521] Step 4:

[0522] The server uses an emotion analysis engine to analyze the emotions of speakers during a meeting and evaluate whether certain phrases indicate anxiety or excitement.

[0523] input:

[0524] Classified text data

[0525] output:

[0526] Emotional data

[0527] Specific actions:

[0528] The server feeds the speech data into an emotion analysis engine, which analyzes whether the "sales manager" felt "anxious" when "reporting sales."

[0529] Step 5:

[0530] The server automatically generates meeting minutes using pre-configured templates based on classified and sentiment-analyzed text data.

[0531] input:

[0532] Classified and sentiment-analyzed text data

[0533] output:

[0534] Automatically generated meeting minutes

[0535] Specific actions:

[0536] The server uses text data and sentiment data to generate formatted meeting minutes.

[0537] Step 6:

[0538] The server saves the generated meeting minutes to the company's database and sends a notification to the user via an HTTP POST request.

[0539] input:

[0540] Automatically generated meeting minutes

[0541] output:

[0542] Saved to database.

[0543] Notification to the user

[0544] Specific actions:

[0545] The server saves the meeting minutes to the company's database and sends a completion notification to the user via email. The user receives and reviews the meeting minutes.

[0546] Automatic question and answer processing system

[0547] Step 1:

[0548] Users can type questions into a text field or speak them aloud during the meeting. The device records these questions and sends them to the server.

[0549] input:

[0550] User's question (text or voice)

[0551] output:

[0552] Recorded question data

[0553] Specific actions:

[0554] The user enters the question "What are next month's sales targets?" into the terminal. The terminal records the question and sends it to the server.

[0555] Step 2:

[0556] If a question is received via voice, the server uses the Google Speech-to-Text API to convert the speech to text.

[0557] input:

[0558] Audio question data

[0559] output:

[0560] Text-based question data

[0561] Specific actions:

[0562] The server converts the audio "What are next month's sales targets?" into text.

[0563] Step 3:

[0564] The server uses the text-based question to issue SQL queries to search for relevant information in the company's database.

[0565] input:

[0566] Text-based question data

[0567] SQL query

[0568] output:

[0569] Related information as search results

[0570] Specific actions:

[0571] The server searches the company database for data related to "next month's sales targets" and retrieves relevant information.

[0572] Step 4:

[0573] The server uses an emotion analysis engine to analyze the questioner's emotions and evaluate whether they are showing dissatisfaction or interest when asking the question.

[0574] input:

[0575] Text-based question data

[0576] output:

[0577] Emotional data

[0578] Specific actions:

[0579] The server uses an emotion analysis engine to confirm that the user expressed dissatisfaction when asking the question.

[0580] Step 5:

[0581] The server automatically generates responses based on the information it acquires, and adjusts the tone and content of the emails based on sentiment data.

[0582] input:

[0583] Related Information

[0584] Emotional data

[0585] output:

[0586] Automated response

[0587] Specific actions:

[0588] The server generates a polite response to the "sales target for next month" and drafts an email in a tone that mitigates any dissatisfaction.

[0589] Step 6:

[0590] The server saves the generated email draft to the company's mail server and sends a completion notification to the user via an HTTP POST request. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[0591] input:

[0592] Automated email draft

[0593] output:

[0594] Saved to mail server.

[0595] Notification to the user

[0596] Specific actions:

[0597] The server saves the generated email draft to the company's mail server and sends a completion notification to the user. The user reviews the draft and makes revisions as needed.

[0598] (Application Example 2)

[0599] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0600] Conventional meeting support systems can automate pre-meeting material preparation, minute-taking, and Q&A processing, but they fail to consider users' feelings towards these materials and minutes. Furthermore, in the unique environment of a factory, there is a particular need for meeting efficiency and user stress reduction. As a result, the quality of meetings can decline, making it difficult to improve user satisfaction.

[0601] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0602] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the retrieved information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for acquiring audio data and performing sentiment analysis, and means for adjusting the content and tone of the automatically generated document based on the results of the sentiment analysis. This makes it possible to provide personalized materials and minutes that take into account the user's emotions in each process of a meeting, such as preparing materials, creating minutes, and answering questions.

[0603] "Schedule data" refers to meeting and event schedules managed by companies and organizations.

[0604] A "performance database" is a database that compiles information about a company's or organization's past performance and achievements.

[0605] An "information database" is a database that aggregates various types of information held by a company or organization.

[0606] "Automatic document generation" is the process of automatically creating documents based on acquired data.

[0607] "Audio data" refers to data that records audio information in digital format.

[0608] "Sentiment analysis" is the process of analyzing the emotions contained in text and audio data using machine learning and natural language processing techniques.

[0609] "User" refers to an individual or organization that uses the system.

[0610] "Related information" refers to useful information related to a specific meeting or event.

[0611] "Storage" refers to the act of storing generated data and documents in a digital format.

[0612] "Notifications" are a means of informing users of important information or events.

[0613] "Speaker" refers to the person who speaks at a meeting or event.

[0614] "Classification" is the process of grouping data according to specific criteria.

[0615] "Meeting minutes" are documents that record the content of discussions and decisions made at meetings or events.

[0616] "Text conversion" is the process of converting audio data into text data.

[0617] "Question content" refers to the specific text or audio of the questions presented at a meeting or event.

[0618] The specific system for realizing this invention includes the following process: First, the server retrieves schedule data and searches for relevant information from the performance database and information database. Next, it integrates the retrieved information and automatically generates a document in a specified format. The generated document is saved and notified to the user.

[0619] The system also includes a function to acquire voice data and perform sentiment analysis. The server collects voice data and analyzes it using a sentiment analysis engine. Based on the results of the sentiment analysis, the content and tone of automatically generated documents are adjusted to provide users with more personalized information.

[0620] To achieve this, the following hardware and software will be used:

[0621] Hardware: Microphone, computer for processing conference audio

[0622] Software: Python, speech_recognition library, nltk library, TextBlob library, gTTS library

[0623] Specific example

[0624] Let's take the example of preparing for a sales meeting to be held next week at a manufacturing plant. The server accesses the company's schedule management system to check the meeting schedule. Furthermore, it retrieves past sales data and new product information from the performance database. Based on this, the server automatically generates preliminary materials in PowerPoint format and adjusts the tone using sentiment analysis. For example, if a user felt stressed during a past meeting, the content of the materials will be adjusted to be more positive.

[0625] Once the meeting begins, the terminal collects audio data in real time and sends it to the server. The server uses speech recognition technology to convert the audio data into text and an emotion analysis engine to analyze the speaker's emotions. Based on the analysis results, the content and tone of the meeting minutes are adjusted and provided to the user.

[0626] Example of a prompt

[0627] Convert the audio file content into text and perform sentiment analysis based on the content. Additionally, retrieve past performance data and product information from relevant databases and generate preliminary materials based on this information.

[0628] This enables highly personalized meeting support that takes emotions into account, reducing user stress and improving meeting efficiency.

[0629] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0630] Step 1:

[0631] The server accesses the company's schedule management system to retrieve the meeting schedule for the following day. For example, it confirms that a "sales meeting" is scheduled. This schedule data is used as input. Based on this data, the server prepares to retrieve information from the performance database and information database required in the next step. The output is schedule information such as the meeting name and date.

[0632] Step 2:

[0633] The server searches the performance database and information database based on the acquired schedule data to find information related to the meeting. Examples include past sales data and new product information. This information is acquired in text format and used for document generation in the next step. The input is schedule data, and the output is text data of related information.

[0634] Step 3:

[0635] The server integrates relevant information and automatically generates documents in a specified format (e.g., PowerPoint or PDF). This document generation process utilizes previously acquired sales data and new product information. Specifically, documents are created by embedding data into a template. The input is text data of the relevant information, and the output is an automatically generated document file.

[0636] Step 4:

[0637] The server saves the document to cloud storage and sends a completion notification to the user. The user receives this notification and can download the document from cloud storage. The input is an automatically generated document file, and the output is a notification message and the document saved in the cloud.

[0638] Step 5:

[0639] The terminal collects audio data from the meeting in real time and streams it to the server. This audio data includes all statements made during the meeting and is used for text conversion and sentiment analysis in the next step. The input is the meeting audio, and the output is the streaming transmission of audio data to the server.

[0640] Step 6:

[0641] The server converts the received audio data into text using a speech recognition algorithm. For example, the audio "It's time for the sales report" is transcribed as "It's time for the sales report." The input is audio data, and the output is text data.

[0642] Step 7:

[0643] The server analyzes the text data and classifies it by speaker. This classification is based on the content of each statement and is organized by speaker name, such as "Sales Manager" or "Technical Manager." The input is text data, and the output is the classified text data.

[0644] Step 8:

[0645] The server uses an emotion engine to analyze the sentiment of each statement in the text data. For example, it analyzes whether a statement is positive or negative, or whether it indicates stress or satisfaction. The input is classified text data, and the output is text data containing sentiment data.

[0646] Step 9:

[0647] The server automatically generates meeting minutes in a specified format based on classified text data and sentiment data. Because it includes sentiment data, the minutes are adjusted to draw attention to specific sections. The input is text data including sentiment data, and the output is automatically generated meeting minutes.

[0648] Step 10:

[0649] The server saves the generated meeting minutes to the database and sends a completion notification to the user. The user receives this notification and reviews the contents of the meeting minutes. The input is the automatically generated meeting minutes, and the output is the notification message and the meeting minutes saved in the database.

[0650] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0651] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0652] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0653] [Second Embodiment]

[0654] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0655] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0656] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0657] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0658] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0659] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0660] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0661] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0662] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0663] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0664] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0665] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0666] Pre-document creation system

[0667] This invention is a system for improving the efficiency of corporate meetings. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, and efficient processing of questions and answers.

[0668] Implementation Example of Preliminary Document Preparation

[0669] 1. Obtaining schedule data

[0670] The server accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, the server confirms that there is a "sales meeting" scheduled.

[0671] 2. Searching for related data

[0672] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[0673] 3. Automatic document generation

[0674] The server integrates the searched data and automatically creates preliminary materials in specified formats such as PowerPoint and PDF. This allows users to prepare high-quality materials without any extra effort.

[0675] 4. Notifications and saving

[0676] The server saves the completed preliminary documents to the company's document management system and sends a notification to the user. The user checks the notification, downloads the documents, and uses them.

[0677] Automatic meeting minutes creation system

[0678] 1. Collection of audio data

[0679] The terminal records audio data during the meeting in real time and sends it to the server.

[0680] 2. Speech Recognition and Text Conversion

[0681] The server receives the audio data and converts it to text using speech recognition technology. For example, it records the sales manager's statement, "It's time for the sales report," as text.

[0682] 3. Text Classification and Analysis

[0683] The server analyzes the text data and categorizes it by speaker, content, and topic. This automatically organizes the meeting minutes.

[0684] 4. Automatic generation and notification of meeting minutes

[0685] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are documented and saved in the specified format. The user receives a notification that the minutes are complete and reviews their contents.

[0686] Automatic question and answer processing system

[0687] 1. Question collection and text transcription

[0688] Users can type or speak questions during the meeting. The device records the questions and, in the case of spoken questions, converts them to text using speech recognition technology.

[0689] 2. Searching for related information

[0690] The server searches the company portal and databases based on the text-based questions and retrieves relevant information.

[0691] 3. Automatic response generation and email draft creation

[0692] The server automatically generates an answer based on the information it has obtained and creates a draft email. The email draft includes the question and the automatically generated answer.

[0693] 4. Notification and Confirmation

[0694] The server saves the completed email draft and sends a notification to the user. The user receives the notification, reviews the email draft, makes any necessary revisions, and sends it.

[0695] As a concrete example, in sales meetings, automating tasks such as reviewing sales performance, introducing new products, and handling questions and answers during the meeting significantly reduces the user's burden and improves productivity. This allows companies to conduct meetings efficiently and make quick and accurate decisions.

[0696] The following describes the processing flow.

[0697] Processing of the program for creating preliminary documents

[0698] Step 1:

[0699] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[0700] Step 2:

[0701] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[0702] Step 3:

[0703] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[0704] Step 4:

[0705] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[0706] Step 5:

[0707] The server integrates the acquired performance data and company profile data, and automatically generates preliminary materials in a specified format (e.g., PowerPoint).

[0708] Step 6:

[0709] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user.

[0710] Processing of the program for automatically creating meeting minutes

[0711] Step 1:

[0712] The terminal records audio data during the meeting in real time and streams the data to the server.

[0713] Step 2:

[0714] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[0715] Step 3:

[0716] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[0717] Step 4:

[0718] The server automatically generates meeting minutes in a pre-configured format based on the classified text data.

[0719] Step 5:

[0720] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user.

[0721] Program processing for automated question and answer session

[0722] Step 1:

[0723] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[0724] Step 2:

[0725] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[0726] Step 3:

[0727] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[0728] Step 4:

[0729] The server automatically generates an answer based on the search results and creates an email draft. The draft includes the question and the answer.

[0730] Step 5:

[0731] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user reviews the draft and makes revisions as needed.

[0732] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, thereby reducing the burden on employees.

[0733] (Example 1)

[0734] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0735] Traditional meeting management methods presented significant challenges, including the time and effort required for everything from scheduling and preparing materials to taking minutes and handling questions during the meeting. In particular, the inability to automate these processes led to decreased corporate efficiency and a lack of speed in decision-making. Furthermore, the manual collection and organization of information by individual staff members was prone to errors and mistakes.

[0736] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0737] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for searching the company portal and database based on the transcribed question content, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This enables the automation and efficiency of the entire meeting management process.

[0738] "Schedule data" refers to information including the scheduled dates and times of meetings and tasks, and is used as the basis for various management and coordination activities.

[0739] A "performance database" is a database that stores data on actual achievements, such as past performance and activity results.

[0740] An "information database" is a database used to store and manage various types of information necessary for business operations, such as product information and customer information.

[0741] "Methods for automatically generating documents" refer to technologies and methods that automatically create documents by integrating related data in a specified format (e.g., PDF or PowerPoint).

[0742] "Means of notifying users" refers to technologies and methods for informing users about generated documents and related information through email or in-application notifications.

[0743] "Audio data" refers to the data format of audio information recorded during meetings or other similar events.

[0744] "Means of transmitting audio data to a server in real time" refers to technologies and methods for sending audio collected during meetings, etc., to a server in real time.

[0745] "Means for converting audio data into text" refers to speech recognition technologies and methods for converting recorded audio into textual information.

[0746] "Text data" refers to character information converted from speech, as well as other text-based data.

[0747] "Methods for analyzing text data and classifying it by speaker" refers to technologies and methods that analyze text data converted from speech and classify and organize information for each speaker.

[0748] "Methods for automatically generating meeting minutes" refers to technologies and methods for automatically creating meeting minutes based on collected and organized text data.

[0749] "Question content" refers to information including questions and concerns raised during the meeting.

[0750] An "internal company portal" is a portal site or system that provides access to information and resources shared within a company.

[0751] "Means of generating answers" refers to technologies and methods that automatically create appropriate answers based on the content of a question.

[0752] A "draft email" is a draft of an email that has been prepared before sending, containing the necessary content but requiring final review and revisions.

[0753] Modes for carrying out the invention

[0754] This invention is a system for improving the efficiency of corporate meetings, providing automatic creation of pre-meeting materials, automatic creation of meeting minutes, and streamlined question-and-answer processing. This system consists of a server, terminals, and users, each performing a specific function.

[0755] Implementation Example of Preliminary Document Preparation

[0756] The server first accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, it might use the Google Calendar API. Through this API, it retrieves the schedule data in JSON format.

[0757] Next, the server uses the acquired schedule data to search for relevant information from the company's performance database and information database. This allows it to retrieve past sales performance and product information. For example, it can use SQL Server or MongoDB to retrieve the necessary data from each database via queries.

[0758] To integrate the searched data and automatically create preliminary materials in specified formats such as PowerPoint and PDF, the server uses libraries such as "python-pptx" and "ReportLab". This allows for the automatic generation of materials by displaying performance data in graphs and placing product information in tables and text boxes.

[0759] The completed documents are saved to the document management system using the SharePoint API. The server then sends a notification to the user. Notification methods include email using an SMTP server and in-app notifications via Microsoft Teams.

[0760] Automatic meeting minutes creation system

[0761] The terminal (a device with a microphone in the conference room) records audio data in real time during the meeting and sends it to the server. The audio data is captured via a "sound card" and streamed to the server using WebSocket.

[0762] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text. This text data is returned in JSON format.

[0763] Next, the server uses a natural language processing library such as "SpaCy" to analyze the text data and classify it by speaker. After classification, it automatically generates meeting minutes based on the classified text data.

[0764] The server uses the Google Docs API to document the meeting minutes and saves the generated minutes to Google Drive. The user is then notified of the URL of the meeting minutes.

[0765] Automatic question and answer processing system

[0766] Users can type questions via their device or speak them aloud during the meeting. If they speak, the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[0767] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[0768] Based on the acquired information, the server automatically generates a response using a generative AI model (e.g., "GPT-3") and creates an email draft using the "Gmail API". The email draft is saved as a draft.

[0769] Finally, a notification is sent to the user, who can review the draft, make any necessary revisions, and send the email.

[0770] Specific example

[0771] For example, in a sales meeting, by inputting the following prompt into the AI ​​model, it is possible to understand the specific steps for creating preliminary materials.

[0772] Example of a prompt:

[0773] For next week's sales meeting, please automatically generate a PowerPoint presentation containing sales performance data for the past six months and information on new products.

[0774] This prompt indicates how the process will proceed. The server will use the Google Calendar API to retrieve the schedule, search for data in SQL Server and MongoDB, and generate a document using python-pptx. Finally, it will use the SharePoint API to save the document and notify the user.

[0775] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0776] System program processing flow

[0777] Implementation Example of Preliminary Document Preparation

[0778] Step 1:

[0779] The server uses the Google Calendar API to retrieve the company's meeting schedule for the following day. This involves the server sending a request to the API and receiving schedule data in JSON format as a response. For example, it might confirm that a "sales meeting" is scheduled for 2:00 PM the following day.

[0780] Input: Request to the Google Calendar API

[0781] Output: Schedule data in JSON format

[0782] Specific operation: Parse JSON data and extract meeting type, date and time, and participant information.

[0783] Step 2:

[0784] Based on the acquired schedule data, the server searches for relevant data such as past sales performance and product information from SQL Server and MongoDB. This allows the server to execute SQL queries to retrieve the necessary performance data and MongoDB queries to retrieve product information.

[0785] Input: Schedule data

[0786] Output: Search results from the performance database and information database.

[0787] Specific operation: Issue specific queries to SQL Server and MongoDB and collect relevant data.

[0788] Step 3:

[0789] The server uses libraries such as "python-pptx" and "ReportLab" to automatically generate preliminary materials in PowerPoint and PDF formats from the integrated data. The server displays past sales performance as graphs and places new product information as tables and text on the slides.

[0790] Input: Performance data and product information

[0791] Output: Preliminary materials in PowerPoint or PDF format

[0792] Specific actions: Format the data and generate slides according to the specified template.

[0793] Step 4:

[0794] The server saves the generated documents to the company's document management system using the SharePoint API. The server then sends a completion notification to the user.

[0795] Input: Generated preliminary data

[0796] Output: URL of documents stored in SharePoint and user notification

[0797] Specific actions: Upload documents to SharePoint and send email notifications via an SMTP server.

[0798] Automatic meeting minutes creation system

[0799] Step 1:

[0800] The terminal records audio data during the meeting in real time and sends it to the server. The terminal captures the audio via a "sound card" and streams the audio data to the server using WebSocket.

[0801] Input: Real-time audio data

[0802] Output: Audio data sent to the server

[0803] Specific operation: Captures audio input from the microphone and transfers the data via a WebSocket connection.

[0804] Step 2:

[0805] The server uses the Google Cloud Speech-to-Text API to convert received audio data into text. It sends audio data to the API and receives text data as a response.

[0806] Input: Audio data

[0807] Output: Text data

[0808] Specific operation: Divide the audio data into fixed batches, send them to the API, and retrieve the conversion results.

[0809] Step 3:

[0810] The server uses "SpaCy" to analyze text data and classify it by speaker. It extracts speaker names and topics from the text data and categorizes them accordingly.

[0811] Input: Text data

[0812] Output: Classified text data

[0813] Specific operation: Performs text analysis, extracts and classifies speaker names and keywords.

[0814] Step 4:

[0815] The server automatically generates meeting minutes based on text data categorized using the Google Docs API. The generated meeting minutes are saved to Google Drive.

[0816] Input: Classified text data

[0817] Output: Meeting minutes saved to Google Drive

[0818] Specific actions: Insert data into a template, generate a document, and save it.

[0819] Step 5:

[0820] The server will notify the user of the URL of the generated meeting minutes. Notification methods include email and in-app notifications.

[0821] Input: URL of the generated meeting minutes

[0822] Output: Completion notification to the user

[0823] Specific operation: Use an SMTP server to send email notifications and generate in-app notifications.

[0824] Automatic question and answer processing system

[0825] Step 1:

[0826] Users can type or speak questions during the meeting. The device records the questions and, if spoken, converts them to text using the Google Cloud Speech-to-Text API.

[0827] Input: Question in voice or text format

[0828] Output: Text version of the question content

[0829] Specific operation: The user either types their speech or captures the audio and converts it to text.

[0830] Step 2:

[0831] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[0832] Input: Text-based question content

[0833] Output: Search Results

[0834] Specific operation: Send a query to Elasticsearch and retrieve relevant information.

[0835] Step 3:

[0836] Based on the acquired information, the server automatically generates answers using a generative AI model (e.g., "GPT-3").

[0837] Input: Search Results

[0838] Output: Auto-generated answer

[0839] Specific operation: Input prompts into the generative AI model and generate responses.

[0840] Step 4:

[0841] The server uses the Gmail API to generate the response, then creates and saves an email draft.

[0842] Input: Auto-generated answer

[0843] Output: Email draft

[0844] Specific operation: Create and save an email draft using the Gmail API.

[0845] Step 5:

[0846] The user is notified of the completed email draft. The user reviews the draft, makes any necessary revisions, and sends the email.

[0847] Input: Email draft

[0848] Output: Completion notification and correction / sent email to the user.

[0849] Specific operation: An SMTP server is used to send a notification, and the user sends an email after making corrections.

[0850] (Application Example 1)

[0851] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0852] In modern factories, meetings and work reports cover a wide range of topics, requiring significant time and effort for preparation and execution. In particular, creating preliminary materials, preparing meeting minutes, and responding quickly to questions are laborious tasks. This situation leads to decreased meeting efficiency and hinders productivity improvements. A system is needed to address these challenges and streamline meetings and work reports within factories.

[0853] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0854] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a performance database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for recording audio data during a meeting in real time and sending it to the server, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for converting the content of questions entered during the meeting into text and searching the company portal and database, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, means for saving the created email draft and notifying the user, and means for generating answers based on prompt sentences using a generation AI model. As a result, the creation of pre-meeting materials, the creation of meeting minutes, and the processing of questions and answers are automated, enabling efficient meeting progress and rapid decision-making.

[0855] "Schedule data" refers to data that shows the planned dates for meetings and tasks.

[0856] A "performance database" is a database that stores records of past work and meetings.

[0857] An "information database" is a database that stores various kinds of related information.

[0858] "Specified format" refers to a standard that specifically instructs the format and layout of a document.

[0859] A "document" refers to a digital document or paper-based material that organizes information according to a specific format.

[0860] "Audio data" refers to data that includes audio waveform data and its digitized form.

[0861] "Text data" refers to data obtained by converting audio data into written text.

[0862] "Speakers" refer to individuals or groups who make statements during meetings or discussions.

[0863] "Meeting minutes" are documents that record the content of meetings or discussions.

[0864] "Questions" refers to the content of questions submitted during meetings or discussions.

[0865] An "internal portal" is an information sharing platform used within a company.

[0866] A "database" is a system that systematically stores and manages data in a searchable format.

[0867] A "draft email" is a draft of an email document before it is sent.

[0868] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to perform data processing or document generation.

[0869] A "prompt sentence" is a sentence containing instructions or questions that are input into a generative AI model.

[0870] This invention is a system aimed at improving the efficiency of meetings and work reports within a factory, and a specific embodiment thereof is shown below. This system includes a server, terminals, and user operation.

[0871] server

[0872] The server uses the following hardware and software:

[0873] Hardware: High-performance processor, ample memory, SSD storage

[0874] Software: Schedule management system, performance database, information database, natural language processing engine, speech recognition engine

[0875] The server first accesses the company's schedule management system and retrieves schedule data. Based on the retrieved schedule data, it searches for relevant information from the performance database and information database, integrates this information, and automatically generates preliminary materials in the specified format (e.g., PDF or PowerPoint). These generated materials are stored in the company's document management system and notified to the user.

[0876] Furthermore, the server receives audio data transmitted from terminals during the meeting in real time and converts it into text using a speech recognition engine. The converted text data is analyzed by a natural language processing engine and classified by speaker. Based on the classified text data, meeting minutes are automatically generated, saved, and then notified to the user.

[0877] Furthermore, the system transcribes questions entered during meetings into text, searches the company portal and databases to retrieve relevant information, and automatically generates answers based on that information, creating a draft email. This draft email is saved and the user is notified. The system also has a function where the generation AI model generates answers based on prompt text.

[0878] terminal

[0879] The devices are smartphones, tablets, or factory robots used within the factory. They are used as follows:

[0880] Collection of audio data

[0881] Record of questions entered during the meeting

[0882] Receipt and confirmation of notification

[0883] The terminal records audio data from the meeting in real time and sends it to the server. Users can also use the terminal to input questions during the meeting. The terminal receives notifications and informs users when materials or meeting minutes are complete.

[0884] User

[0885] The user performs the following actions:

[0886] Receive notifications from the system

[0887] Review the prepared preliminary documents and make any necessary corrections.

[0888] Review the automatically generated meeting minutes and make corrections if necessary.

[0889] Review the draft email, make any necessary corrections, and then send it.

[0890] Specific example

[0891] As a concrete example, the following prompt statements are used as input to the generating AI model:

[0892] Prompt: Based on the schedule data and past performance data, please prepare the meeting materials for the following day.

[0893] data:

[0894] Schedule: Next day's schedule: Safety meeting

[0895] Track record: Past safety reports

[0896] In this way, the creation of preliminary materials, the preparation of meeting minutes, and the processing of questions and answers are automated, significantly improving the efficiency of meetings and work reports within the factory.

[0897] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0898] Step 1:

[0899] The server accesses the company's schedule management system and retrieves schedule data. This retrieved schedule data becomes the input for the current process. Based on this data, the server proceeds to the next processing step.

[0900] Step 2:

[0901] The server searches for relevant information from the performance database and information database based on the acquired schedule data. The results of the search, including performance data and related information, are output. This output is used to create preliminary documents.

[0902] Step 3:

[0903] The server integrates the performance data and related information obtained in Step 2 and automatically generates a document in the specified format. Specifically, it outputs the document as a PDF or PowerPoint file. This output document is used as preliminary material.

[0904] Step 4:

[0905] The server saves the generated document to the company's document management system and sends a notification to the user when saving is complete. The user receives the notification and downloads and reviews the document as needed.

[0906] Step 5:

[0907] The terminal records audio data during the meeting in real time and sends that data to the server. The audio data arrives at the server as input and proceeds to the next processing step.

[0908] Step 6:

[0909] The server converts the transmitted audio data into text data using a speech recognition engine. This converted text data is then output and analyzed.

[0910] Step 7:

[0911] The server analyzes the converted text data using a natural language processing engine and classifies it by speaker. Text data is used as input, and the classified text data is obtained as output.

[0912] Step 8:

[0913] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are output in a specific format and stored in the company's document management system.

[0914] Step 9:

[0915] The server sends a notification to the user when the generated meeting minutes have been saved. The user receives the notification and reviews the minutes, making any necessary corrections.

[0916] Step 10:

[0917] The system transcribes questions entered by users on their devices during meetings into text and sends this text data to a server. The server analyzes this text data and searches the company portal and databases to retrieve relevant information.

[0918] Step 11:

[0919] The server automatically generates a response using a generative AI model based on the acquired relevant information. The prompt text and relevant information are used as input, and text data is output as the response.

[0920] Step 12:

[0921] The server creates an email draft based on the generated response, saves this draft, and notifies the user. The user receives the notification, reviews the email content as needed, makes any necessary corrections, and then sends it.

[0922] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0923] Pre-document creation system

[0924] This invention is a system for improving the efficiency of corporate meetings, and is characterized by its ability to generate more personalized materials, meeting minutes, and Q&A responses by taking user emotions into consideration. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, efficient processing of questions and answers, and adjustments by an emotion engine.

[0925] Implementation Example of Preliminary Document Preparation

[0926] 1. Obtaining schedule data

[0927] The server accesses the company's schedule management system to check the meeting schedule for the next day. For example, it might find that there is a meeting named "Sales Meeting".

[0928] 2. Searching for related data

[0929] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[0930] 3. Emotion recognition by an emotion engine

[0931] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[0932] 4. Automatic document generation and adjustment

[0933] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[0934] 5. Notifications and saving

[0935] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[0936] Automatic meeting minutes creation system

[0937] 1. Collection of audio data

[0938] The terminal records audio data during the meeting in real time and streams it to the server.

[0939] 2. Speech Recognition and Text Conversion

[0940] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[0941] 3. Text Classification and Analysis

[0942] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[0943] 4. Analysis using an emotional engine

[0944] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[0945] 5. Automatic generation and adjustment of meeting minutes

[0946] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[0947] 6. Saving and Notifications

[0948] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[0949] Automatic question and answer processing system

[0950] 1. Question collection and text transcription

[0951] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[0952] 2. Text conversion using speech recognition

[0953] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[0954] 3. Searching for related information

[0955] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[0956] 4. Feedback from the Emotion Engine

[0957] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[0958] 5. Automatic response generation and email draft creation

[0959] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[0960] 6. Notification and Confirmation

[0961] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[0962] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, and provide advanced personalization that takes user emotions into account, thereby reducing the burden on employees and improving productivity.

[0963] The following describes the processing flow.

[0964] Processing of the program for creating preliminary documents

[0965] Step 1:

[0966] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[0967] Step 2:

[0968] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[0969] Step 3:

[0970] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[0971] Step 4:

[0972] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[0973] Step 5:

[0974] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[0975] Step 6:

[0976] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[0977] Step 7:

[0978] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[0979] Processing of the program for automatically creating meeting minutes

[0980] Step 1:

[0981] The terminal records audio data during the meeting in real time and streams the data to the server.

[0982] Step 2:

[0983] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[0984] Step 3:

[0985] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[0986] Step 4:

[0987] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[0988] Step 5:

[0989] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[0990] Step 6:

[0991] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[0992] Program processing for automated question and answer session

[0993] Step 1:

[0994] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[0995] Step 2:

[0996] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[0997] Step 3:

[0998] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[0999] Step 4:

[1000] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[1001] Step 5:

[1002] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[1003] Step 6:

[1004] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[1005] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, provide advanced personalization that takes user emotions into account, reduce employee workload, and improve productivity.

[1006] (Example 2)

[1007] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1008] Traditional meeting management systems often require manual processes for meeting preparation, minute-taking, and Q&A, resulting in significant time and effort. Furthermore, these tasks often fail to consider user emotions, impacting meeting efficiency and participant satisfaction. Therefore, there is a need for a system that automates meeting preparation, recording, and Q&A, while also providing personalized responses that take user emotions into account.

[1009] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1010] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a history database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for adjusting the representation of the data using an emotion analysis engine when generating the document, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for analyzing emotions based on the text data, means for automatically generating meeting minutes based on the classified and emotion-analyzed text data, means for searching an internal portal and a database based on the transcribed question content, means for automatically generating answers based on the search results, means for adjusting the representation of the answers using an emotion analysis engine when generating the answers, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This automates meeting preparation, recording, and question handling, and enables advanced personalization that takes user emotions into consideration.

[1011] "Schedule data" refers to data that includes a user's schedule and appointments. It is obtained from the company's schedule management system.

[1012] A "history database" refers to a database that stores data on past achievements and activities.

[1013] An "information database" refers to a database that stores data such as product information and company policies.

[1014] An "emotion analysis engine" refers to software or a system that analyzes a user's emotions from text or audio and outputs the results as emotion data.

[1015] "Documents" refer to digital files containing organized and edited information such as meeting materials, minutes, and email drafts.

[1016] "Specified format" refers to the predetermined format and layout used when creating a document.

[1017] "Audio data" refers to data that digitally saves the content of participants' remarks recorded during a meeting.

[1018] "Text" refers to data obtained by converting audio data into written information.

[1019] "Classification" refers to dividing speakers and their statements into specific categories based on text data.

[1020] "Meeting minutes" refers to a document that records the content and statements made during a meeting.

[1021] An "internal portal" refers to a web-based platform used for information sharing and communication within a company.

[1022] "Answer" refers to the information provided as a response to a question.

[1023] A "draft email" refers to the content of an email that has been saved in its state before being sent to the user.

[1024] This invention relates to a system that automates meeting preparation, minute-taking, and question-and-answer processing, and further generates more personalized materials, minutes, and Q&A by taking user emotions into consideration. This system mainly consists of a server, terminals, and users, and is implemented using the following specific hardware and software.

[1025] Pre-document creation system

[1026] 1. Hardware and software configuration

[1027] The server accesses a schedule management system (e.g., Google Calendar) to retrieve data. Python is the primary programming language, and MySQL and PostgreSQL are used as databases.

[1028] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions and use that information to adjust the content of the materials.

[1029] The server uses a combination of Python libraries to automatically generate pre-recorded materials in PowerPoint or PDF format (e.g., python-pptx, ReportLab).

[1030] 2. Specific Examples

[1031] The server retrieves the schedule data for the next day's "sales meeting" and searches the database for past sales performance data and the latest product information. Using an emotion analysis engine, it automatically generates pre-meeting materials that heavily utilize positive language, based on data showing that the user exhibited high stress levels in past sales meetings.

[1032] Example prompts for generative AI models

[1033] Please prepare the preliminary materials for the next day's "Sales Meeting" using the following information and maintaining a positive tone.

[1034] September sales: ¥10,000,000

[1035] New product XYZ specification update

[1036] To reduce the user's past stress

[1037] Automatic meeting minutes creation system

[1038] 1. Hardware and software configuration

[1039] The terminal uses its microphone to record audio data during the meeting and utilizes WebSocket to stream it to the server in real time.

[1040] The server uses the Google Speech-to-Text API to convert the audio data into text.

[1041] The server uses natural language processing tools (e.g., NLTK) to analyze text data and classify it by speaker. It also uses a sentiment analysis engine to analyze emotions.

[1042] 2. Specific Examples

[1043] The terminal records the audio during the meeting and sends it to the server in real time. The server converts the audio, such as "It's time for the sales report," into text, classifies the speaker and the content of their statement, and also performs sentiment analysis.

[1044] Example prompts for generative AI models

[1045] Please create meeting minutes that include the following audio data. Please pay attention to the emotions of the speakers.

[1046] Sales Manager: "It's time for the sales report."

[1047] Emotional data: "Anxiety"

[1048] Automatic question and answer processing system

[1049] 1. Hardware and software configuration

[1050] Users can type questions into their devices or speak them aloud during the meeting. The devices record these questions and send them to the server.

[1051] The server uses the Google Speech-to-Text API to convert voice questions into text and then uses SQL queries to search for relevant information in the company's database.

[1052] The server uses an emotion analysis engine to analyze the questioner's emotions and adjusts the tone of the response accordingly.

[1053] 2. Specific Examples

[1054] If a user asks "What are next month's sales targets?" during a meeting, the device records the question and sends it to the server. The server converts the audio to text, searches for relevant information, and generates a polite response based on sentiment analysis.

[1055] Example prompts for generative AI models

[1056] Please generate answers to the following questions in a polite tone.

[1057] Question text: "What are your sales targets for next month?"

[1058] Emotional data: "Dissatisfaction"

[1059] Through the system configuration and processing steps described above, this invention automates meeting preparation, recording, and question handling, while also providing a high level of personalization that takes user emotions into consideration. This is expected to reduce the burden on employees and improve productivity.

[1060] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1061] Pre-document creation system

[1062] Step 1:

[1063] The server sends an HTTP request to the company's scheduling management system (e.g., Google Calendar) and retrieves the next day's meeting schedule data in JSON format.

[1064] input:

[1065] API endpoint for the schedule management system

[1066] HTTP Request

[1067] output:

[1068] Retrieved schedule data (JSON format)

[1069] Specific actions:

[1070] The server sends an HTTP request and receives schedule data in JSON format. For example, it retrieves data for "October 11, 2023, 9:00 - 10:00 Sales Meeting".

[1071] Step 2:

[1072] Based on the retrieved schedule data, the server issues SQL queries to search for the necessary data from the history database (e.g., MySQL) and the information database (e.g., PostgreSQL).

[1073] input:

[1074] Acquired schedule data

[1075] SQL query

[1076] output:

[1077] Search results include historical data and product information.

[1078] Specific actions:

[1079] Based on the acquired schedule data, the server issues an SQL query to retrieve historical data such as "Sales in September 2023: ¥10,000,000" and product information such as "Specification update for new product XYZ".

[1080] Step 3:

[1081] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the emotion data the user expressed in previous meetings.

[1082] input:

[1083] Past meeting data (audio or text)

[1084] output:

[1085] Emotional data (categories and scores such as stress, satisfaction, and anxiety)

[1086] Specific actions:

[1087] The server feeds audio data from past sales meetings into an emotion analysis engine and obtains analysis results indicating that the user showed high stress levels during past meetings.

[1088] Step 4:

[1089] The server integrates the acquired historical data and product information, and uses a Python script to automatically generate preliminary materials in specified formats such as PowerPoint and PDF.

[1090] input:

[1091] Historical data

[1092] Product Information

[1093] Emotional data

[1094] output:

[1095] Automatically generated pre-document materials (PowerPoint or PDF format)

[1096] Specific actions:

[1097] The server integrates historical data and product information, and creates preliminary materials in PowerPoint format, adjusting the tone of the materials based on sentiment data. For example, it reduces user stress by adding positive language.

[1098] Step 5:

[1099] The server uploads the generated documents to the company's cloud storage (e.g., Google Drive) and sends a completion notification to the user via an HTTP POST request.

[1100] input:

[1101] Automatically generated preliminary materials

[1102] Cloud storage API endpoints

[1103] output:

[1104] Upload to cloud storage complete.

[1105] Notification to the user

[1106] Specific actions:

[1107] The server uploads the created preliminary documents to Google Drive and sends a completion notification to the user via email. The user receives the email, downloads the documents, and reviews them.

[1108] Automatic meeting minutes creation system

[1109] Step 1:

[1110] The terminal uses its microphone to record audio data during the meeting and streams it to the server in real time via WebSocket.

[1111] input:

[1112] Meeting audio

[1113] output:

[1114] Streamed audio data

[1115] Specific actions:

[1116] The device picks up all audio during the meeting using its microphone and transmits it to the server in real time.

[1117] Step 2:

[1118] The server sends the received audio data to the Google Speech-to-Text API to convert the audio into text.

[1119] input:

[1120] Audio data

[1121] output:

[1122] Text data

[1123] Specific actions:

[1124] The server converts the audio "It's time for the sales report" into text "It's time for the sales report".

[1125] Step 3:

[1126] The server analyzes the acquired text and classifies the data based on factors such as the speaker's name, the content of the statement, and the topic. It uses natural language processing tools (e.g., NLTK).

[1127] input:

[1128] Text data

[1129] output:

[1130] Classified text data

[1131] Specific actions:

[1132] The server analyzes the text data and categorizes it as "Sales Manager" or "Sales Report."

[1133] Step 4:

[1134] The server uses an emotion analysis engine to analyze the emotions of speakers during a meeting and evaluate whether certain phrases indicate anxiety or excitement.

[1135] input:

[1136] Classified text data

[1137] output:

[1138] Emotional data

[1139] Specific actions:

[1140] The server feeds the speech data into an emotion analysis engine, which analyzes whether the "sales manager" felt "anxious" when "reporting sales."

[1141] Step 5:

[1142] The server automatically generates meeting minutes using pre-configured templates based on classified and sentiment-analyzed text data.

[1143] input:

[1144] Classified and sentiment-analyzed text data

[1145] output:

[1146] Automatically generated meeting minutes

[1147] Specific actions:

[1148] The server uses text data and sentiment data to generate formatted meeting minutes.

[1149] Step 6:

[1150] The server saves the generated meeting minutes to the company's database and sends a notification to the user via an HTTP POST request.

[1151] input:

[1152] Automatically generated meeting minutes

[1153] output:

[1154] Saved to database.

[1155] Notification to the user

[1156] Specific actions:

[1157] The server saves the meeting minutes to the company's database and sends a completion notification to the user via email. The user receives and reviews the meeting minutes.

[1158] Automatic question and answer processing system

[1159] Step 1:

[1160] Users can type questions into a text field or speak them aloud during the meeting. The device records these questions and sends them to the server.

[1161] input:

[1162] User's question (text or voice)

[1163] output:

[1164] Recorded question data

[1165] Specific actions:

[1166] The user enters the question "What are next month's sales targets?" into the terminal. The terminal records the question and sends it to the server.

[1167] Step 2:

[1168] If a question is received via voice, the server uses the Google Speech-to-Text API to convert the speech to text.

[1169] input:

[1170] Audio question data

[1171] output:

[1172] Text-based question data

[1173] Specific actions:

[1174] The server converts the audio "What are next month's sales targets?" into text.

[1175] Step 3:

[1176] The server uses the text-based question to issue SQL queries to search for relevant information in the company's database.

[1177] input:

[1178] Text-based question data

[1179] SQL query

[1180] output:

[1181] Related information as search results

[1182] Specific actions:

[1183] The server searches the company database for data related to "next month's sales targets" and retrieves relevant information.

[1184] Step 4:

[1185] The server uses an emotion analysis engine to analyze the questioner's emotions and evaluate whether they are showing dissatisfaction or interest when asking the question.

[1186] input:

[1187] Text-based question data

[1188] output:

[1189] Emotional data

[1190] Specific actions:

[1191] The server uses an emotion analysis engine to confirm that the user expressed dissatisfaction when asking the question.

[1192] Step 5:

[1193] The server automatically generates responses based on the information it acquires, and adjusts the tone and content of the emails based on sentiment data.

[1194] input:

[1195] Related Information

[1196] Emotional data

[1197] output:

[1198] Automated response

[1199] Specific actions:

[1200] The server generates a polite response to the "sales target for next month" and drafts an email in a tone that mitigates any dissatisfaction.

[1201] Step 6:

[1202] The server saves the generated email draft to the company's mail server and sends a completion notification to the user via an HTTP POST request. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[1203] input:

[1204] Automated email draft

[1205] output:

[1206] Saved to mail server.

[1207] Notification to the user

[1208] Specific actions:

[1209] The server saves the generated email draft to the company's mail server and sends a completion notification to the user. The user reviews the draft and makes revisions as needed.

[1210] (Application Example 2)

[1211] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1212] Conventional meeting support systems can automate pre-meeting material preparation, minute-taking, and Q&A processing, but they fail to consider users' feelings towards these materials and minutes. Furthermore, in the unique environment of a factory, there is a particular need for meeting efficiency and user stress reduction. As a result, the quality of meetings can decline, making it difficult to improve user satisfaction.

[1213] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1214] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the retrieved information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for acquiring audio data and performing sentiment analysis, and means for adjusting the content and tone of the automatically generated document based on the results of the sentiment analysis. This makes it possible to provide personalized materials and minutes that take into account the user's emotions in each process of a meeting, such as preparing materials, creating minutes, and answering questions.

[1215] "Schedule data" refers to meeting and event schedules managed by companies and organizations.

[1216] A "performance database" is a database that compiles information about a company's or organization's past performance and achievements.

[1217] An "information database" is a database that aggregates various types of information held by a company or organization.

[1218] "Automatic document generation" is the process of automatically creating documents based on acquired data.

[1219] "Audio data" refers to data that records audio information in digital format.

[1220] "Sentiment analysis" is the process of analyzing the emotions contained in text and audio data using machine learning and natural language processing techniques.

[1221] "User" refers to an individual or organization that uses the system.

[1222] "Related information" refers to useful information related to a specific meeting or event.

[1223] "Storage" refers to the act of storing generated data and documents in a digital format.

[1224] "Notifications" are a means of informing users of important information or events.

[1225] "Speaker" refers to the person who speaks at a meeting or event.

[1226] "Classification" is the process of grouping data according to specific criteria.

[1227] "Meeting minutes" are documents that record the content of discussions and decisions made at meetings or events.

[1228] "Text conversion" is the process of converting audio data into text data.

[1229] "Question content" refers to the specific text or audio of the questions presented at a meeting or event.

[1230] The specific system for realizing this invention includes the following process: First, the server retrieves schedule data and searches for relevant information from the performance database and information database. Next, it integrates the retrieved information and automatically generates a document in a specified format. The generated document is saved and notified to the user.

[1231] The system also includes a function to acquire voice data and perform sentiment analysis. The server collects voice data and analyzes it using a sentiment analysis engine. Based on the results of the sentiment analysis, the content and tone of automatically generated documents are adjusted to provide users with more personalized information.

[1232] To achieve this, the following hardware and software will be used:

[1233] Hardware: Microphone, computer for processing conference audio

[1234] Software: Python, speech_recognition library, nltk library, TextBlob library, gTTS library

[1235] Specific example

[1236] Let's take the example of preparing for a sales meeting to be held next week at a manufacturing plant. The server accesses the company's schedule management system to check the meeting schedule. Furthermore, it retrieves past sales data and new product information from the performance database. Based on this, the server automatically generates preliminary materials in PowerPoint format and adjusts the tone using sentiment analysis. For example, if a user felt stressed during a past meeting, the content of the materials will be adjusted to be more positive.

[1237] Once the meeting begins, the terminal collects audio data in real time and sends it to the server. The server uses speech recognition technology to convert the audio data into text and an emotion analysis engine to analyze the speaker's emotions. Based on the analysis results, the content and tone of the meeting minutes are adjusted and provided to the user.

[1238] Example of a prompt

[1239] Convert the audio file content into text and perform sentiment analysis based on the content. Additionally, retrieve past performance data and product information from relevant databases and generate preliminary materials based on this information.

[1240] This enables highly personalized meeting support that takes emotions into account, reducing user stress and improving meeting efficiency.

[1241] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1242] Step 1:

[1243] The server accesses the company's schedule management system to retrieve the meeting schedule for the following day. For example, it confirms that a "sales meeting" is scheduled. This schedule data is used as input. Based on this data, the server prepares to retrieve information from the performance database and information database required in the next step. The output is schedule information such as the meeting name and date.

[1244] Step 2:

[1245] The server searches the performance database and information database based on the acquired schedule data to find information related to the meeting. Examples include past sales data and new product information. This information is acquired in text format and used for document generation in the next step. The input is schedule data, and the output is text data of related information.

[1246] Step 3:

[1247] The server integrates relevant information and automatically generates documents in a specified format (e.g., PowerPoint or PDF). This document generation process utilizes previously acquired sales data and new product information. Specifically, documents are created by embedding data into a template. The input is text data of the relevant information, and the output is an automatically generated document file.

[1248] Step 4:

[1249] The server saves the document to cloud storage and sends a completion notification to the user. The user receives this notification and can download the document from cloud storage. The input is an automatically generated document file, and the output is a notification message and the document saved in the cloud.

[1250] Step 5:

[1251] The terminal collects audio data from the meeting in real time and streams it to the server. This audio data includes all statements made during the meeting and is used for text conversion and sentiment analysis in the next step. The input is the meeting audio, and the output is the streaming transmission of audio data to the server.

[1252] Step 6:

[1253] The server converts the received audio data into text using a speech recognition algorithm. For example, the audio "It's time for the sales report" is transcribed as "It's time for the sales report." The input is audio data, and the output is text data.

[1254] Step 7:

[1255] The server analyzes the text data and classifies it by speaker. This classification is based on the content of each statement and is organized by speaker name, such as "Sales Manager" or "Technical Manager." The input is text data, and the output is the classified text data.

[1256] Step 8:

[1257] The server uses an emotion engine to analyze the sentiment of each statement in the text data. For example, it analyzes whether a statement is positive or negative, or whether it indicates stress or satisfaction. The input is classified text data, and the output is text data containing sentiment data.

[1258] Step 9:

[1259] The server automatically generates meeting minutes in a specified format based on classified text data and sentiment data. Because it includes sentiment data, the minutes are adjusted to draw attention to specific sections. The input is text data including sentiment data, and the output is automatically generated meeting minutes.

[1260] Step 10:

[1261] The server saves the generated meeting minutes to the database and sends a completion notification to the user. The user receives this notification and reviews the contents of the meeting minutes. The input is the automatically generated meeting minutes, and the output is the notification message and the meeting minutes saved in the database.

[1262] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1263] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1264] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1265] [Third Embodiment]

[1266] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1267] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1268] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1269] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1270] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1271] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1272] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1273] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1274] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1275] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1276] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1277] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1278] Pre-document creation system

[1279] This invention is a system for improving the efficiency of corporate meetings. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, and efficient processing of questions and answers.

[1280] Implementation Example of Preliminary Document Preparation

[1281] 1. Obtaining schedule data

[1282] The server accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, the server confirms that there is a "sales meeting" scheduled.

[1283] 2. Searching for related data

[1284] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[1285] 3. Automatic document generation

[1286] The server integrates the searched data and automatically creates preliminary materials in specified formats such as PowerPoint and PDF. This allows users to prepare high-quality materials without any extra effort.

[1287] 4. Notifications and saving

[1288] The server saves the completed preliminary documents to the company's document management system and sends a notification to the user. The user checks the notification, downloads the documents, and uses them.

[1289] Automatic meeting minutes creation system

[1290] 1. Collection of audio data

[1291] The terminal records audio data during the meeting in real time and sends it to the server.

[1292] 2. Speech Recognition and Text Conversion

[1293] The server receives the audio data and converts it to text using speech recognition technology. For example, it records the sales manager's statement, "It's time for the sales report," as text.

[1294] 3. Text Classification and Analysis

[1295] The server analyzes the text data and categorizes it by speaker, content, and topic. This automatically organizes the meeting minutes.

[1296] 4. Automatic generation and notification of meeting minutes

[1297] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are documented and saved in the specified format. The user receives a notification that the minutes are complete and reviews their contents.

[1298] Automatic question and answer processing system

[1299] 1. Question collection and text transcription

[1300] Users can type or speak questions during the meeting. The device records the questions and, in the case of spoken questions, converts them to text using speech recognition technology.

[1301] 2. Searching for related information

[1302] The server searches the company portal and databases based on the text-based questions and retrieves relevant information.

[1303] 3. Automatic response generation and email draft creation

[1304] The server automatically generates an answer based on the information it has obtained and creates a draft email. The email draft includes the question and the automatically generated answer.

[1305] 4. Notification and Confirmation

[1306] The server saves the completed email draft and sends a notification to the user. The user receives the notification, reviews the email draft, makes any necessary revisions, and sends it.

[1307] As a concrete example, in sales meetings, automating tasks such as reviewing sales performance, introducing new products, and handling questions and answers during the meeting significantly reduces the user's burden and improves productivity. This allows companies to conduct meetings efficiently and make quick and accurate decisions.

[1308] The following describes the processing flow.

[1309] Processing of the program for creating preliminary documents

[1310] Step 1:

[1311] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[1312] Step 2:

[1313] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[1314] Step 3:

[1315] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[1316] Step 4:

[1317] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[1318] Step 5:

[1319] The server integrates the acquired performance data and company profile data, and automatically generates preliminary materials in a specified format (e.g., PowerPoint).

[1320] Step 6:

[1321] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user.

[1322] Processing of the program for automatically creating meeting minutes

[1323] Step 1:

[1324] The terminal records audio data during the meeting in real time and streams the data to the server.

[1325] Step 2:

[1326] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[1327] Step 3:

[1328] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[1329] Step 4:

[1330] The server automatically generates meeting minutes in a pre-configured format based on the classified text data.

[1331] Step 5:

[1332] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user.

[1333] Program processing for automated question and answer session

[1334] Step 1:

[1335] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[1336] Step 2:

[1337] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[1338] Step 3:

[1339] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[1340] Step 4:

[1341] The server automatically generates an answer based on the search results and creates an email draft. The draft includes the question and the answer.

[1342] Step 5:

[1343] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user reviews the draft and makes revisions as needed.

[1344] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, thereby reducing the burden on employees.

[1345] (Example 1)

[1346] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1347] Traditional meeting management methods presented significant challenges, including the time and effort required for everything from scheduling and preparing materials to taking minutes and handling questions during the meeting. In particular, the inability to automate these processes led to decreased corporate efficiency and a lack of speed in decision-making. Furthermore, the manual collection and organization of information by individual staff members was prone to errors and mistakes.

[1348] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1349] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for searching the company portal and database based on the transcribed question content, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This enables the automation and efficiency of the entire meeting management process.

[1350] "Schedule data" refers to information including the scheduled dates and times of meetings and tasks, and is used as the basis for various management and coordination activities.

[1351] A "performance database" is a database that stores data on actual achievements, such as past performance and activity results.

[1352] An "information database" is a database used to store and manage various types of information necessary for business operations, such as product information and customer information.

[1353] "Methods for automatically generating documents" refer to technologies and methods that automatically create documents by integrating related data in a specified format (e.g., PDF or PowerPoint).

[1354] "Means of notifying users" refers to technologies and methods for informing users about generated documents and related information through email or in-application notifications.

[1355] "Audio data" refers to the data format of audio information recorded during meetings or other similar events.

[1356] "Means of transmitting audio data to a server in real time" refers to technologies and methods for sending audio collected during meetings, etc., to a server in real time.

[1357] "Means for converting audio data into text" refers to speech recognition technologies and methods for converting recorded audio into textual information.

[1358] "Text data" refers to character information converted from speech, as well as other text-based data.

[1359] "Methods for analyzing text data and classifying it by speaker" refers to technologies and methods that analyze text data converted from speech and classify and organize information for each speaker.

[1360] "Methods for automatically generating meeting minutes" refers to technologies and methods for automatically creating meeting minutes based on collected and organized text data.

[1361] "Question content" refers to information including questions and concerns raised during the meeting.

[1362] An "internal company portal" is a portal site or system that provides access to information and resources shared within a company.

[1363] "Means of generating answers" refers to technologies and methods that automatically create appropriate answers based on the content of a question.

[1364] A "draft email" is a draft of an email that has been prepared before sending, containing the necessary content but requiring final review and revisions.

[1365] Modes for carrying out the invention

[1366] This invention is a system for improving the efficiency of corporate meetings, providing automatic creation of pre-meeting materials, automatic creation of meeting minutes, and streamlined question-and-answer processing. This system consists of a server, terminals, and users, each performing a specific function.

[1367] Implementation Example of Preliminary Document Preparation

[1368] The server first accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, it might use the Google Calendar API. Through this API, it retrieves the schedule data in JSON format.

[1369] Next, the server uses the acquired schedule data to search for relevant information from the company's performance database and information database. This allows it to retrieve past sales performance and product information. For example, it can use SQL Server or MongoDB to retrieve the necessary data from each database via queries.

[1370] To integrate the searched data and automatically create preliminary materials in specified formats such as PowerPoint and PDF, the server uses libraries such as "python-pptx" and "ReportLab". This allows for the automatic generation of materials by displaying performance data in graphs and placing product information in tables and text boxes.

[1371] The completed documents are saved to the document management system using the SharePoint API. The server then sends a notification to the user. Notification methods include email using an SMTP server and in-app notifications via Microsoft Teams.

[1372] Automatic meeting minutes creation system

[1373] The terminal (a device with a microphone in the conference room) records audio data in real time during the meeting and sends it to the server. The audio data is captured via a "sound card" and streamed to the server using WebSocket.

[1374] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text. This text data is returned in JSON format.

[1375] Next, the server uses a natural language processing library such as "SpaCy" to analyze the text data and classify it by speaker. After classification, it automatically generates meeting minutes based on the classified text data.

[1376] The server uses the Google Docs API to document the meeting minutes and saves the generated minutes to Google Drive. The user is then notified of the URL of the meeting minutes.

[1377] Automatic question and answer processing system

[1378] Users can type questions via their device or speak them aloud during the meeting. If they speak, the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[1379] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[1380] Based on the acquired information, the server automatically generates a response using a generative AI model (e.g., "GPT-3") and creates an email draft using the "Gmail API". The email draft is saved as a draft.

[1381] Finally, a notification is sent to the user, who can review the draft, make any necessary revisions, and send the email.

[1382] Specific example

[1383] For example, in a sales meeting, by inputting the following prompt into the AI ​​model, it is possible to understand the specific steps for creating preliminary materials.

[1384] Example of a prompt:

[1385] For next week's sales meeting, please automatically generate a PowerPoint presentation containing sales performance data for the past six months and information on new products.

[1386] This prompt indicates how the process will proceed. The server will use the Google Calendar API to retrieve the schedule, search for data in SQL Server and MongoDB, and generate a document using python-pptx. Finally, it will use the SharePoint API to save the document and notify the user.

[1387] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1388] System program processing flow

[1389] Implementation Example of Preliminary Document Preparation

[1390] Step 1:

[1391] The server uses the Google Calendar API to retrieve the company's meeting schedule for the following day. This involves the server sending a request to the API and receiving schedule data in JSON format as a response. For example, it might confirm that a "sales meeting" is scheduled for 2:00 PM the following day.

[1392] Input: Request to the Google Calendar API

[1393] Output: Schedule data in JSON format

[1394] Specific operation: Parse JSON data and extract meeting type, date and time, and participant information.

[1395] Step 2:

[1396] Based on the acquired schedule data, the server searches for relevant data such as past sales performance and product information from SQL Server and MongoDB. This allows the server to execute SQL queries to retrieve the necessary performance data and MongoDB queries to retrieve product information.

[1397] Input: Schedule data

[1398] Output: Search results from the performance database and information database.

[1399] Specific operation: Issue specific queries to SQL Server and MongoDB and collect relevant data.

[1400] Step 3:

[1401] The server uses libraries such as "python-pptx" and "ReportLab" to automatically generate preliminary materials in PowerPoint and PDF formats from the integrated data. The server displays past sales performance as graphs and places new product information as tables and text on the slides.

[1402] Input: Performance data and product information

[1403] Output: Preliminary materials in PowerPoint or PDF format

[1404] Specific actions: Format the data and generate slides according to the specified template.

[1405] Step 4:

[1406] The server saves the generated documents to the company's document management system using the SharePoint API. The server then sends a completion notification to the user.

[1407] Input: Generated preliminary data

[1408] Output: URL of documents stored in SharePoint and user notification

[1409] Specific actions: Upload documents to SharePoint and send email notifications via an SMTP server.

[1410] Automatic meeting minutes creation system

[1411] Step 1:

[1412] The terminal records audio data during the meeting in real time and sends it to the server. The terminal captures the audio via a "sound card" and streams the audio data to the server using WebSocket.

[1413] Input: Real-time audio data

[1414] Output: Audio data sent to the server

[1415] Specific operation: Captures audio input from the microphone and transfers the data via a WebSocket connection.

[1416] Step 2:

[1417] The server uses the Google Cloud Speech-to-Text API to convert received audio data into text. It sends audio data to the API and receives text data as a response.

[1418] Input: Audio data

[1419] Output: Text data

[1420] Specific operation: Divide the audio data into fixed batches, send them to the API, and retrieve the conversion results.

[1421] Step 3:

[1422] The server uses "SpaCy" to analyze text data and classify it by speaker. It extracts speaker names and topics from the text data and categorizes them accordingly.

[1423] Input: Text data

[1424] Output: Classified text data

[1425] Specific operation: Performs text analysis, extracts and classifies speaker names and keywords.

[1426] Step 4:

[1427] The server automatically generates meeting minutes based on text data categorized using the Google Docs API. The generated meeting minutes are saved to Google Drive.

[1428] Input: Classified text data

[1429] Output: Meeting minutes saved to Google Drive

[1430] Specific actions: Insert data into a template, generate a document, and save it.

[1431] Step 5:

[1432] The server will notify the user of the URL of the generated meeting minutes. Notification methods include email and in-app notifications.

[1433] Input: URL of the generated meeting minutes

[1434] Output: Completion notification to the user

[1435] Specific operation: Use an SMTP server to send email notifications and generate in-app notifications.

[1436] Automatic question and answer processing system

[1437] Step 1:

[1438] Users can type or speak questions during the meeting. The device records the questions and, if spoken, converts them to text using the Google Cloud Speech-to-Text API.

[1439] Input: Question in voice or text format

[1440] Output: Text version of the question content

[1441] Specific operation: The user either types their speech or captures the audio and converts it to text.

[1442] Step 2:

[1443] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[1444] Input: Text-based question content

[1445] Output: Search Results

[1446] Specific operation: Send a query to Elasticsearch and retrieve relevant information.

[1447] Step 3:

[1448] Based on the acquired information, the server automatically generates answers using a generative AI model (e.g., "GPT-3").

[1449] Input: Search Results

[1450] Output: Auto-generated answer

[1451] Specific operation: Input prompts into the generative AI model and generate responses.

[1452] Step 4:

[1453] The server uses the Gmail API to generate the response, then creates and saves an email draft.

[1454] Input: Auto-generated answer

[1455] Output: Email draft

[1456] Specific operation: Create and save an email draft using the Gmail API.

[1457] Step 5:

[1458] The user is notified of the completed email draft. The user reviews the draft, makes any necessary revisions, and sends the email.

[1459] Input: Email draft

[1460] Output: Completion notification and correction / sent email to the user.

[1461] Specific operation: An SMTP server is used to send a notification, and the user sends an email after making corrections.

[1462] (Application Example 1)

[1463] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1464] In modern factories, meetings and work reports cover a wide range of topics, requiring significant time and effort for preparation and execution. In particular, creating preliminary materials, preparing meeting minutes, and responding quickly to questions are laborious tasks. This situation leads to decreased meeting efficiency and hinders productivity improvements. A system is needed to address these challenges and streamline meetings and work reports within factories.

[1465] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1466] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a performance database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for recording audio data during a meeting in real time and sending it to the server, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for converting the content of questions entered during the meeting into text and searching the company portal and database, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, means for saving the created email draft and notifying the user, and means for generating answers based on prompt sentences using a generation AI model. As a result, the creation of pre-meeting materials, the creation of meeting minutes, and the processing of questions and answers are automated, enabling efficient meeting progress and rapid decision-making.

[1467] "Schedule data" refers to data that shows the planned dates for meetings and tasks.

[1468] A "performance database" is a database that stores records of past work and meetings.

[1469] An "information database" is a database that stores various kinds of related information.

[1470] "Specified format" refers to a standard that specifically instructs the format and layout of a document.

[1471] A "document" refers to a digital document or paper-based material that organizes information according to a specific format.

[1472] "Audio data" refers to data that includes audio waveform data and its digitized form.

[1473] "Text data" refers to data obtained by converting audio data into written text.

[1474] "Speakers" refer to individuals or groups who make statements during meetings or discussions.

[1475] "Meeting minutes" are documents that record the content of meetings or discussions.

[1476] "Questions" refers to the content of questions submitted during meetings or discussions.

[1477] An "internal portal" is an information sharing platform used within a company.

[1478] A "database" is a system that systematically stores and manages data in a searchable format.

[1479] A "draft email" is a draft of an email document before it is sent.

[1480] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to perform data processing or document generation.

[1481] A "prompt sentence" is a sentence containing instructions or questions that are input into a generative AI model.

[1482] This invention is a system aimed at improving the efficiency of meetings and work reports within a factory, and a specific embodiment thereof is shown below. This system includes a server, terminals, and user operation.

[1483] server

[1484] The server uses the following hardware and software:

[1485] Hardware: High-performance processor, ample memory, SSD storage

[1486] Software: Schedule management system, performance database, information database, natural language processing engine, speech recognition engine

[1487] The server first accesses the company's schedule management system and retrieves schedule data. Based on the retrieved schedule data, it searches for relevant information from the performance database and information database, integrates this information, and automatically generates preliminary materials in the specified format (e.g., PDF or PowerPoint). These generated materials are stored in the company's document management system and notified to the user.

[1488] Furthermore, the server receives audio data transmitted from terminals during the meeting in real time and converts it into text using a speech recognition engine. The converted text data is analyzed by a natural language processing engine and classified by speaker. Based on the classified text data, meeting minutes are automatically generated, saved, and then notified to the user.

[1489] Furthermore, the system transcribes questions entered during meetings into text, searches the company portal and databases to retrieve relevant information, and automatically generates answers based on that information, creating a draft email. This draft email is saved and the user is notified. The system also has a function where the generation AI model generates answers based on prompt text.

[1490] terminal

[1491] The devices are smartphones, tablets, or factory robots used within the factory. They are used as follows:

[1492] Collection of audio data

[1493] Record of questions entered during the meeting

[1494] Receipt and confirmation of notification

[1495] The terminal records audio data from the meeting in real time and sends it to the server. Users can also use the terminal to input questions during the meeting. The terminal receives notifications and informs users when materials or meeting minutes are complete.

[1496] User

[1497] The user performs the following actions:

[1498] Receive notifications from the system

[1499] Review the prepared preliminary documents and make any necessary corrections.

[1500] Review the automatically generated meeting minutes and make corrections if necessary.

[1501] Review the draft email, make any necessary corrections, and then send it.

[1502] Specific example

[1503] As a concrete example, the following prompt statements are used as input to the generating AI model:

[1504] Prompt: Based on the schedule data and past performance data, please prepare the meeting materials for the following day.

[1505] data:

[1506] Schedule: Next day's schedule: Safety meeting

[1507] Track record: Past safety reports

[1508] In this way, the creation of preliminary materials, the preparation of meeting minutes, and the processing of questions and answers are automated, significantly improving the efficiency of meetings and work reports within the factory.

[1509] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1510] Step 1:

[1511] The server accesses the company's schedule management system and retrieves schedule data. This retrieved schedule data becomes the input for the current process. Based on this data, the server proceeds to the next processing step.

[1512] Step 2:

[1513] The server searches for relevant information from the performance database and information database based on the acquired schedule data. The results of the search, including performance data and related information, are output. This output is used to create preliminary documents.

[1514] Step 3:

[1515] The server integrates the performance data and related information obtained in Step 2 and automatically generates a document in the specified format. Specifically, it outputs the document as a PDF or PowerPoint file. This output document is used as preliminary material.

[1516] Step 4:

[1517] The server saves the generated document to the company's document management system and sends a notification to the user when saving is complete. The user receives the notification and downloads and reviews the document as needed.

[1518] Step 5:

[1519] The terminal records audio data during the meeting in real time and sends that data to the server. The audio data arrives at the server as input and proceeds to the next processing step.

[1520] Step 6:

[1521] The server converts the transmitted audio data into text data using a speech recognition engine. This converted text data is then output and analyzed.

[1522] Step 7:

[1523] The server analyzes the converted text data using a natural language processing engine and classifies it by speaker. Text data is used as input, and the classified text data is obtained as output.

[1524] Step 8:

[1525] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are output in a specific format and stored in the company's document management system.

[1526] Step 9:

[1527] The server sends a notification to the user when the generated meeting minutes have been saved. The user receives the notification and reviews the minutes, making any necessary corrections.

[1528] Step 10:

[1529] The system transcribes questions entered by users on their devices during meetings into text and sends this text data to a server. The server analyzes this text data and searches the company portal and databases to retrieve relevant information.

[1530] Step 11:

[1531] The server automatically generates a response using a generative AI model based on the acquired relevant information. The prompt text and relevant information are used as input, and text data is output as the response.

[1532] Step 12:

[1533] The server creates an email draft based on the generated response, saves this draft, and notifies the user. The user receives the notification, reviews the email content as needed, makes any necessary corrections, and then sends it.

[1534] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1535] Pre-document creation system

[1536] This invention is a system for improving the efficiency of corporate meetings, and is characterized by its ability to generate more personalized materials, meeting minutes, and Q&A responses by taking user emotions into consideration. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, efficient processing of questions and answers, and adjustments by an emotion engine.

[1537] Implementation Example of Preliminary Document Preparation

[1538] 1. Obtaining schedule data

[1539] The server accesses the company's schedule management system to check the meeting schedule for the next day. For example, it might find that there is a meeting named "Sales Meeting".

[1540] 2. Searching for related data

[1541] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[1542] 3. Emotion recognition by an emotion engine

[1543] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[1544] 4. Automatic document generation and adjustment

[1545] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[1546] 5. Notifications and saving

[1547] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[1548] Automatic meeting minutes creation system

[1549] 1. Collection of audio data

[1550] The terminal records audio data during the meeting in real time and streams it to the server.

[1551] 2. Speech Recognition and Text Conversion

[1552] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[1553] 3. Text Classification and Analysis

[1554] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[1555] 4. Analysis using an emotional engine

[1556] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[1557] 5. Automatic generation and adjustment of meeting minutes

[1558] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[1559] 6. Saving and Notifications

[1560] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[1561] Automatic question and answer processing system

[1562] 1. Question collection and text transcription

[1563] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[1564] 2. Text conversion using speech recognition

[1565] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[1566] 3. Searching for related information

[1567] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[1568] 4. Feedback from the Emotion Engine

[1569] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[1570] 5. Automatic response generation and email draft creation

[1571] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[1572] 6. Notification and Confirmation

[1573] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[1574] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, and provide advanced personalization that takes user emotions into account, thereby reducing the burden on employees and improving productivity.

[1575] The following describes the processing flow.

[1576] Processing of the program for creating preliminary documents

[1577] Step 1:

[1578] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[1579] Step 2:

[1580] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[1581] Step 3:

[1582] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[1583] Step 4:

[1584] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[1585] Step 5:

[1586] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[1587] Step 6:

[1588] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[1589] Step 7:

[1590] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[1591] Processing of the program for automatically creating meeting minutes

[1592] Step 1:

[1593] The terminal records audio data during the meeting in real time and streams the data to the server.

[1594] Step 2:

[1595] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[1596] Step 3:

[1597] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[1598] Step 4:

[1599] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[1600] Step 5:

[1601] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[1602] Step 6:

[1603] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[1604] Program processing for automated question and answer session

[1605] Step 1:

[1606] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[1607] Step 2:

[1608] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[1609] Step 3:

[1610] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[1611] Step 4:

[1612] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[1613] Step 5:

[1614] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[1615] Step 6:

[1616] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[1617] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, provide advanced personalization that takes user emotions into account, reduce employee workload, and improve productivity.

[1618] (Example 2)

[1619] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1620] Traditional meeting management systems often require manual processes for meeting preparation, minute-taking, and Q&A, resulting in significant time and effort. Furthermore, these tasks often fail to consider user emotions, impacting meeting efficiency and participant satisfaction. Therefore, there is a need for a system that automates meeting preparation, recording, and Q&A, while also providing personalized responses that take user emotions into account.

[1621] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1622] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a history database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for adjusting the representation of the data using an emotion analysis engine when generating the document, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for analyzing emotions based on the text data, means for automatically generating meeting minutes based on the classified and emotion-analyzed text data, means for searching an internal portal and a database based on the transcribed question content, means for automatically generating answers based on the search results, means for adjusting the representation of the answers using an emotion analysis engine when generating the answers, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This automates meeting preparation, recording, and question handling, and enables advanced personalization that takes user emotions into consideration.

[1623] "Schedule data" refers to data that includes a user's schedule and appointments. It is obtained from the company's schedule management system.

[1624] A "history database" refers to a database that stores data on past achievements and activities.

[1625] An "information database" refers to a database that stores data such as product information and company policies.

[1626] An "emotion analysis engine" refers to software or a system that analyzes a user's emotions from text or audio and outputs the results as emotion data.

[1627] "Documents" refer to digital files containing organized and edited information such as meeting materials, minutes, and email drafts.

[1628] "Specified format" refers to the predetermined format and layout used when creating a document.

[1629] "Audio data" refers to data that digitally saves the content of participants' remarks recorded during a meeting.

[1630] "Text" refers to data obtained by converting audio data into written information.

[1631] "Classification" refers to dividing speakers and their statements into specific categories based on text data.

[1632] "Meeting minutes" refers to a document that records the content and statements made during a meeting.

[1633] An "internal portal" refers to a web-based platform used for information sharing and communication within a company.

[1634] "Answer" refers to the information provided as a response to a question.

[1635] A "draft email" refers to the content of an email that has been saved in its state before being sent to the user.

[1636] This invention relates to a system that automates meeting preparation, minute-taking, and question-and-answer processing, and further generates more personalized materials, minutes, and Q&A by taking user emotions into consideration. This system mainly consists of a server, terminals, and users, and is implemented using the following specific hardware and software.

[1637] Pre-document creation system

[1638] 1. Hardware and software configuration

[1639] The server accesses a schedule management system (e.g., Google Calendar) to retrieve data. Python is the primary programming language, and MySQL and PostgreSQL are used as databases.

[1640] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions and use that information to adjust the content of the materials.

[1641] The server uses a combination of Python libraries to automatically generate pre-recorded materials in PowerPoint or PDF format (e.g., python-pptx, ReportLab).

[1642] 2. Specific Examples

[1643] The server retrieves the schedule data for the next day's "sales meeting" and searches the database for past sales performance data and the latest product information. Using an emotion analysis engine, it automatically generates pre-meeting materials that heavily utilize positive language, based on data showing that the user exhibited high stress levels in past sales meetings.

[1644] Example prompts for generative AI models

[1645] Please prepare the preliminary materials for the next day's "Sales Meeting" using the following information and maintaining a positive tone.

[1646] September sales: ¥10,000,000

[1647] New product XYZ specification update

[1648] To reduce the user's past stress

[1649] Automatic meeting minutes creation system

[1650] 1. Hardware and software configuration

[1651] The terminal uses its microphone to record audio data during the meeting and utilizes WebSocket to stream it to the server in real time.

[1652] The server uses the Google Speech-to-Text API to convert the audio data into text.

[1653] The server uses natural language processing tools (e.g., NLTK) to analyze text data and classify it by speaker. It also uses a sentiment analysis engine to analyze emotions.

[1654] 2. Specific Examples

[1655] The terminal records the audio during the meeting and sends it to the server in real time. The server converts the audio, such as "It's time for the sales report," into text, classifies the speaker and the content of their statement, and also performs sentiment analysis.

[1656] Example prompts for generative AI models

[1657] Please create meeting minutes that include the following audio data. Please pay attention to the emotions of the speakers.

[1658] Sales Manager: "It's time for the sales report."

[1659] Emotional data: "Anxiety"

[1660] Automatic question and answer processing system

[1661] 1. Hardware and software configuration

[1662] Users can type questions into their devices or speak them aloud during the meeting. The devices record these questions and send them to the server.

[1663] The server uses the Google Speech-to-Text API to convert voice questions into text and then uses SQL queries to search for relevant information in the company's database.

[1664] The server uses an emotion analysis engine to analyze the questioner's emotions and adjusts the tone of the response accordingly.

[1665] 2. Specific Examples

[1666] If a user asks "What are next month's sales targets?" during a meeting, the device records the question and sends it to the server. The server converts the audio to text, searches for relevant information, and generates a polite response based on sentiment analysis.

[1667] Example prompts for generative AI models

[1668] Please generate answers to the following questions in a polite tone.

[1669] Question text: "What are your sales targets for next month?"

[1670] Emotional data: "Dissatisfaction"

[1671] Through the system configuration and processing steps described above, this invention automates meeting preparation, recording, and question handling, while also providing a high level of personalization that takes user emotions into consideration. This is expected to reduce the burden on employees and improve productivity.

[1672] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1673] Pre-document creation system

[1674] Step 1:

[1675] The server sends an HTTP request to the company's scheduling management system (e.g., Google Calendar) and retrieves the next day's meeting schedule data in JSON format.

[1676] input:

[1677] API endpoint for the schedule management system

[1678] HTTP Request

[1679] output:

[1680] Retrieved schedule data (JSON format)

[1681] Specific actions:

[1682] The server sends an HTTP request and receives schedule data in JSON format. For example, it retrieves data for "October 11, 2023, 9:00 - 10:00 Sales Meeting".

[1683] Step 2:

[1684] Based on the retrieved schedule data, the server issues SQL queries to search for the necessary data from the history database (e.g., MySQL) and the information database (e.g., PostgreSQL).

[1685] input:

[1686] Acquired schedule data

[1687] SQL query

[1688] output:

[1689] Search results include historical data and product information.

[1690] Specific actions:

[1691] Based on the acquired schedule data, the server issues an SQL query to retrieve historical data such as "Sales in September 2023: ¥10,000,000" and product information such as "Specification update for new product XYZ".

[1692] Step 3:

[1693] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the emotion data the user expressed in previous meetings.

[1694] input:

[1695] Past meeting data (audio or text)

[1696] output:

[1697] Emotional data (categories and scores such as stress, satisfaction, and anxiety)

[1698] Specific actions:

[1699] The server feeds audio data from past sales meetings into an emotion analysis engine and obtains analysis results indicating that the user showed high stress levels during past meetings.

[1700] Step 4:

[1701] The server integrates the acquired historical data and product information, and uses a Python script to automatically generate preliminary materials in specified formats such as PowerPoint and PDF.

[1702] input:

[1703] Historical data

[1704] Product Information

[1705] Emotional data

[1706] output:

[1707] Automatically generated pre-document materials (PowerPoint or PDF format)

[1708] Specific actions:

[1709] The server integrates historical data and product information, and creates preliminary materials in PowerPoint format, adjusting the tone of the materials based on sentiment data. For example, it reduces user stress by adding positive language.

[1710] Step 5:

[1711] The server uploads the generated documents to the company's cloud storage (e.g., Google Drive) and sends a completion notification to the user via an HTTP POST request.

[1712] input:

[1713] Automatically generated preliminary materials

[1714] Cloud storage API endpoints

[1715] output:

[1716] Upload to cloud storage complete.

[1717] Notification to the user

[1718] Specific actions:

[1719] The server uploads the created preliminary documents to Google Drive and sends a completion notification to the user via email. The user receives the email, downloads the documents, and reviews them.

[1720] Automatic meeting minutes creation system

[1721] Step 1:

[1722] The terminal uses its microphone to record audio data during the meeting and streams it to the server in real time via WebSocket.

[1723] input:

[1724] Meeting audio

[1725] output:

[1726] Streamed audio data

[1727] Specific actions:

[1728] The device picks up all audio during the meeting using its microphone and transmits it to the server in real time.

[1729] Step 2:

[1730] The server sends the received audio data to the Google Speech-to-Text API to convert the audio into text.

[1731] input:

[1732] Audio data

[1733] output:

[1734] Text data

[1735] Specific actions:

[1736] The server converts the audio "It's time for the sales report" into text "It's time for the sales report".

[1737] Step 3:

[1738] The server analyzes the acquired text and classifies the data based on factors such as the speaker's name, the content of the statement, and the topic. It uses natural language processing tools (e.g., NLTK).

[1739] input:

[1740] Text data

[1741] output:

[1742] Classified text data

[1743] Specific actions:

[1744] The server analyzes the text data and categorizes it as "Sales Manager" or "Sales Report."

[1745] Step 4:

[1746] The server uses an emotion analysis engine to analyze the emotions of speakers during a meeting and evaluate whether certain phrases indicate anxiety or excitement.

[1747] input:

[1748] Classified text data

[1749] output:

[1750] Emotional data

[1751] Specific actions:

[1752] The server feeds the speech data into an emotion analysis engine, which analyzes whether the "sales manager" felt "anxious" when "reporting sales."

[1753] Step 5:

[1754] The server automatically generates meeting minutes using pre-configured templates based on classified and sentiment-analyzed text data.

[1755] input:

[1756] Classified and sentiment-analyzed text data

[1757] output:

[1758] Automatically generated meeting minutes

[1759] Specific actions:

[1760] The server uses text data and sentiment data to generate formatted meeting minutes.

[1761] Step 6:

[1762] The server saves the generated meeting minutes to the company's database and sends a notification to the user via an HTTP POST request.

[1763] input:

[1764] Automatically generated meeting minutes

[1765] output:

[1766] Saved to database.

[1767] Notification to the user

[1768] Specific actions:

[1769] The server saves the meeting minutes to the company's database and sends a completion notification to the user via email. The user receives and reviews the meeting minutes.

[1770] Automatic question and answer processing system

[1771] Step 1:

[1772] Users can type questions into a text field or speak them aloud during the meeting. The device records these questions and sends them to the server.

[1773] input:

[1774] User's question (text or voice)

[1775] output:

[1776] Recorded question data

[1777] Specific actions:

[1778] The user enters the question "What are next month's sales targets?" into the terminal. The terminal records the question and sends it to the server.

[1779] Step 2:

[1780] If a question is received via voice, the server uses the Google Speech-to-Text API to convert the speech to text.

[1781] input:

[1782] Audio question data

[1783] output:

[1784] Text-based question data

[1785] Specific actions:

[1786] The server converts the audio "What are next month's sales targets?" into text.

[1787] Step 3:

[1788] The server uses the text-based question to issue SQL queries to search for relevant information in the company's database.

[1789] input:

[1790] Text-based question data

[1791] SQL query

[1792] output:

[1793] Related information as search results

[1794] Specific actions:

[1795] The server searches the company database for data related to "next month's sales targets" and retrieves relevant information.

[1796] Step 4:

[1797] The server uses an emotion analysis engine to analyze the questioner's emotions and evaluate whether they are showing dissatisfaction or interest when asking the question.

[1798] input:

[1799] Text-based question data

[1800] output:

[1801] Emotional data

[1802] Specific actions:

[1803] The server uses an emotion analysis engine to confirm that the user expressed dissatisfaction when asking the question.

[1804] Step 5:

[1805] The server automatically generates responses based on the information it acquires, and adjusts the tone and content of the emails based on sentiment data.

[1806] input:

[1807] Related Information

[1808] Emotional data

[1809] output:

[1810] Automated response

[1811] Specific actions:

[1812] The server generates a polite response to the "sales target for next month" and drafts an email in a tone that mitigates any dissatisfaction.

[1813] Step 6:

[1814] The server saves the generated email draft to the company's mail server and sends a completion notification to the user via an HTTP POST request. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[1815] input:

[1816] Automated email draft

[1817] output:

[1818] Saved to mail server.

[1819] Notification to the user

[1820] Specific actions:

[1821] The server saves the generated email draft to the company's mail server and sends a completion notification to the user. The user reviews the draft and makes revisions as needed.

[1822] (Application Example 2)

[1823] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1824] Conventional meeting support systems can automate pre-meeting material preparation, minute-taking, and Q&A processing, but they fail to consider users' feelings towards these materials and minutes. Furthermore, in the unique environment of a factory, there is a particular need for meeting efficiency and user stress reduction. As a result, the quality of meetings can decline, making it difficult to improve user satisfaction.

[1825] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1826] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the retrieved information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for acquiring audio data and performing sentiment analysis, and means for adjusting the content and tone of the automatically generated document based on the results of the sentiment analysis. This makes it possible to provide personalized materials and minutes that take into account the user's emotions in each process of a meeting, such as preparing materials, creating minutes, and answering questions.

[1827] "Schedule data" refers to meeting and event schedules managed by companies and organizations.

[1828] A "performance database" is a database that compiles information about a company's or organization's past performance and achievements.

[1829] An "information database" is a database that aggregates various types of information held by a company or organization.

[1830] "Automatic document generation" is the process of automatically creating documents based on acquired data.

[1831] "Audio data" refers to data that records audio information in digital format.

[1832] "Sentiment analysis" is the process of analyzing the emotions contained in text and audio data using machine learning and natural language processing techniques.

[1833] "User" refers to an individual or organization that uses the system.

[1834] "Related information" refers to useful information related to a specific meeting or event.

[1835] "Storage" refers to the act of storing generated data and documents in a digital format.

[1836] "Notifications" are a means of informing users of important information or events.

[1837] "Speaker" refers to the person who speaks at a meeting or event.

[1838] "Classification" is the process of grouping data according to specific criteria.

[1839] "Meeting minutes" are documents that record the content of discussions and decisions made at meetings or events.

[1840] "Text conversion" is the process of converting audio data into text data.

[1841] "Question content" refers to the specific text or audio of the questions presented at a meeting or event.

[1842] The specific system for realizing this invention includes the following process: First, the server retrieves schedule data and searches for relevant information from the performance database and information database. Next, it integrates the retrieved information and automatically generates a document in a specified format. The generated document is saved and notified to the user.

[1843] The system also includes a function to acquire voice data and perform sentiment analysis. The server collects voice data and analyzes it using a sentiment analysis engine. Based on the results of the sentiment analysis, the content and tone of automatically generated documents are adjusted to provide users with more personalized information.

[1844] To achieve this, the following hardware and software will be used:

[1845] Hardware: Microphone, computer for processing conference audio

[1846] Software: Python, speech_recognition library, nltk library, TextBlob library, gTTS library

[1847] Specific example

[1848] Let's take the example of preparing for a sales meeting to be held next week at a manufacturing plant. The server accesses the company's schedule management system to check the meeting schedule. Furthermore, it retrieves past sales data and new product information from the performance database. Based on this, the server automatically generates preliminary materials in PowerPoint format and adjusts the tone using sentiment analysis. For example, if a user felt stressed during a past meeting, the content of the materials will be adjusted to be more positive.

[1849] Once the meeting begins, the terminal collects audio data in real time and sends it to the server. The server uses speech recognition technology to convert the audio data into text and an emotion analysis engine to analyze the speaker's emotions. Based on the analysis results, the content and tone of the meeting minutes are adjusted and provided to the user.

[1850] Example of a prompt

[1851] Convert the audio file content into text and perform sentiment analysis based on the content. Additionally, retrieve past performance data and product information from relevant databases and generate preliminary materials based on this information.

[1852] This enables highly personalized meeting support that takes emotions into account, reducing user stress and improving meeting efficiency.

[1853] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1854] Step 1:

[1855] The server accesses the company's schedule management system to retrieve the meeting schedule for the following day. For example, it confirms that a "sales meeting" is scheduled. This schedule data is used as input. Based on this data, the server prepares to retrieve information from the performance database and information database required in the next step. The output is schedule information such as the meeting name and date.

[1856] Step 2:

[1857] The server searches the performance database and information database based on the acquired schedule data to find information related to the meeting. Examples include past sales data and new product information. This information is acquired in text format and used for document generation in the next step. The input is schedule data, and the output is text data of related information.

[1858] Step 3:

[1859] The server integrates relevant information and automatically generates documents in a specified format (e.g., PowerPoint or PDF). This document generation process utilizes previously acquired sales data and new product information. Specifically, documents are created by embedding data into a template. The input is text data of the relevant information, and the output is an automatically generated document file.

[1860] Step 4:

[1861] The server saves the document to cloud storage and sends a completion notification to the user. The user receives this notification and can download the document from cloud storage. The input is an automatically generated document file, and the output is a notification message and the document saved in the cloud.

[1862] Step 5:

[1863] The terminal collects audio data from the meeting in real time and streams it to the server. This audio data includes all statements made during the meeting and is used for text conversion and sentiment analysis in the next step. The input is the meeting audio, and the output is the streaming transmission of audio data to the server.

[1864] Step 6:

[1865] The server converts the received audio data into text using a speech recognition algorithm. For example, the audio "It's time for the sales report" is transcribed as "It's time for the sales report." The input is audio data, and the output is text data.

[1866] Step 7:

[1867] The server analyzes the text data and classifies it by speaker. This classification is based on the content of each statement and is organized by speaker name, such as "Sales Manager" or "Technical Manager." The input is text data, and the output is the classified text data.

[1868] Step 8:

[1869] The server uses an emotion engine to analyze the sentiment of each statement in the text data. For example, it analyzes whether a statement is positive or negative, or whether it indicates stress or satisfaction. The input is classified text data, and the output is text data containing sentiment data.

[1870] Step 9:

[1871] The server automatically generates meeting minutes in a specified format based on classified text data and sentiment data. Because it includes sentiment data, the minutes are adjusted to draw attention to specific sections. The input is text data including sentiment data, and the output is automatically generated meeting minutes.

[1872] Step 10:

[1873] The server saves the generated meeting minutes to the database and sends a completion notification to the user. The user receives this notification and reviews the contents of the meeting minutes. The input is the automatically generated meeting minutes, and the output is the notification message and the meeting minutes saved in the database.

[1874] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1875] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1876] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1877] [Fourth Embodiment]

[1878] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1879] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1880] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1881] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1882] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1883] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1884] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1885] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1886] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1887] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1888] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1889] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1890] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1891] Pre-document creation system

[1892] This invention is a system for improving the efficiency of corporate meetings. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, and efficient processing of questions and answers.

[1893] Implementation Example of Preliminary Document Preparation

[1894] 1. Obtaining schedule data

[1895] The server accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, the server confirms that there is a "sales meeting" scheduled.

[1896] 2. Searching for related data

[1897] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[1898] 3. Automatic document generation

[1899] The server integrates the searched data and automatically creates preliminary materials in specified formats such as PowerPoint and PDF. This allows users to prepare high-quality materials without any extra effort.

[1900] 4. Notifications and saving

[1901] The server saves the completed preliminary documents to the company's document management system and sends a notification to the user. The user checks the notification, downloads the documents, and uses them.

[1902] Automatic meeting minutes creation system

[1903] 1. Collection of audio data

[1904] The terminal records audio data during the meeting in real time and sends it to the server.

[1905] 2. Speech Recognition and Text Conversion

[1906] The server receives the audio data and converts it to text using speech recognition technology. For example, it records the sales manager's statement, "It's time for the sales report," as text.

[1907] 3. Text Classification and Analysis

[1908] The server analyzes the text data and categorizes it by speaker, content, and topic. This automatically organizes the meeting minutes.

[1909] 4. Automatic generation and notification of meeting minutes

[1910] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are documented and saved in the specified format. The user receives a notification that the minutes are complete and reviews their contents.

[1911] Automatic question and answer processing system

[1912] 1. Question collection and text transcription

[1913] Users can type or speak questions during the meeting. The device records the questions and, in the case of spoken questions, converts them to text using speech recognition technology.

[1914] 2. Searching for related information

[1915] The server searches the company portal and databases based on the text-based questions and retrieves relevant information.

[1916] 3. Automatic response generation and email draft creation

[1917] The server automatically generates an answer based on the information it has obtained and creates a draft email. The email draft includes the question and the automatically generated answer.

[1918] 4. Notification and Confirmation

[1919] The server saves the completed email draft and sends a notification to the user. The user receives the notification, reviews the email draft, makes any necessary revisions, and sends it.

[1920] As a concrete example, in sales meetings, automating tasks such as reviewing sales performance, introducing new products, and handling questions and answers during the meeting significantly reduces the user's burden and improves productivity. This allows companies to conduct meetings efficiently and make quick and accurate decisions.

[1921] The following describes the processing flow.

[1922] Processing of the program for creating preliminary documents

[1923] Step 1:

[1924] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[1925] Step 2:

[1926] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[1927] Step 3:

[1928] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[1929] Step 4:

[1930] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[1931] Step 5:

[1932] The server integrates the acquired performance data and company profile data, and automatically generates preliminary materials in a specified format (e.g., PowerPoint).

[1933] Step 6:

[1934] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user.

[1935] Processing of the program for automatically creating meeting minutes

[1936] Step 1:

[1937] The terminal records audio data during the meeting in real time and streams the data to the server.

[1938] Step 2:

[1939] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[1940] Step 3:

[1941] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[1942] Step 4:

[1943] The server automatically generates meeting minutes in a pre-configured format based on the classified text data.

[1944] Step 5:

[1945] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user.

[1946] Program processing for automated question and answer session

[1947] Step 1:

[1948] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[1949] Step 2:

[1950] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[1951] Step 3:

[1952] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[1953] Step 4:

[1954] The server automatically generates an answer based on the search results and creates an email draft. The draft includes the question and the answer.

[1955] Step 5:

[1956] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user reviews the draft and makes revisions as needed.

[1957] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, thereby reducing the burden on employees.

[1958] (Example 1)

[1959] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1960] Traditional meeting management methods presented significant challenges, including the time and effort required for everything from scheduling and preparing materials to taking minutes and handling questions during the meeting. In particular, the inability to automate these processes led to decreased corporate efficiency and a lack of speed in decision-making. Furthermore, the manual collection and organization of information by individual staff members was prone to errors and mistakes.

[1961] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1962] In this invention, the server includes means for acquiring schedule data, means for searching relevant information from a performance database and an information database, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for searching the company portal and database based on the transcribed question content, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This enables the automation and efficiency of the entire meeting management process.

[1963] "Schedule data" refers to information including the scheduled dates and times of meetings and tasks, and is used as the basis for various management and coordination activities.

[1964] A "performance database" is a database that stores data on actual achievements, such as past performance and activity results.

[1965] An "information database" is a database used to store and manage various types of information necessary for business operations, such as product information and customer information.

[1966] "Methods for automatically generating documents" refer to technologies and methods that automatically create documents by integrating related data in a specified format (e.g., PDF or PowerPoint).

[1967] "Means of notifying users" refers to technologies and methods for informing users about generated documents and related information through email or in-application notifications.

[1968] "Audio data" refers to the data format of audio information recorded during meetings or other similar events.

[1969] "Means of transmitting audio data to a server in real time" refers to technologies and methods for sending audio collected during meetings, etc., to a server in real time.

[1970] "Means for converting audio data into text" refers to speech recognition technologies and methods for converting recorded audio into textual information.

[1971] "Text data" refers to character information converted from speech, as well as other text-based data.

[1972] "Methods for analyzing text data and classifying it by speaker" refers to technologies and methods that analyze text data converted from speech and classify and organize information for each speaker.

[1973] "Methods for automatically generating meeting minutes" refers to technologies and methods for automatically creating meeting minutes based on collected and organized text data.

[1974] "Question content" refers to information including questions and concerns raised during the meeting.

[1975] An "internal company portal" is a portal site or system that provides access to information and resources shared within a company.

[1976] "Means of generating answers" refers to technologies and methods that automatically create appropriate answers based on the content of a question.

[1977] A "draft email" is a draft of an email that has been prepared before sending, containing the necessary content but requiring final review and revisions.

[1978] Modes for carrying out the invention

[1979] This invention is a system for improving the efficiency of corporate meetings, providing automatic creation of pre-meeting materials, automatic creation of meeting minutes, and streamlined question-and-answer processing. This system consists of a server, terminals, and users, each performing a specific function.

[1980] Implementation Example of Preliminary Document Preparation

[1981] The server first accesses the company's scheduling software to retrieve the meeting schedule for the following day. For example, it might use the Google Calendar API. Through this API, it retrieves the schedule data in JSON format.

[1982] Next, the server uses the acquired schedule data to search for relevant information from the company's performance database and information database. This allows it to retrieve past sales performance and product information. For example, it can use SQL Server or MongoDB to retrieve the necessary data from each database via queries.

[1983] To integrate the searched data and automatically create preliminary materials in specified formats such as PowerPoint and PDF, the server uses libraries such as "python-pptx" and "ReportLab". This allows for the automatic generation of materials by displaying performance data in graphs and placing product information in tables and text boxes.

[1984] The completed documents are saved to the document management system using the SharePoint API. The server then sends a notification to the user. Notification methods include email using an SMTP server and in-app notifications via Microsoft Teams.

[1985] Automatic meeting minutes creation system

[1986] The terminal (a device with a microphone in the conference room) records audio data in real time during the meeting and sends it to the server. The audio data is captured via a "sound card" and streamed to the server using WebSocket.

[1987] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text. This text data is returned in JSON format.

[1988] Next, the server uses a natural language processing library such as "SpaCy" to analyze the text data and classify it by speaker. After classification, it automatically generates meeting minutes based on the classified text data.

[1989] The server uses the Google Docs API to document the meeting minutes and saves the generated minutes to Google Drive. The user is then notified of the URL of the meeting minutes.

[1990] Automatic question and answer processing system

[1991] Users can type questions via their device or speak them aloud during the meeting. If they speak, the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[1992] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[1993] Based on the acquired information, the server automatically generates a response using a generative AI model (e.g., "GPT-3") and creates an email draft using the "Gmail API". The email draft is saved as a draft.

[1994] Finally, a notification is sent to the user, who can review the draft, make any necessary revisions, and send the email.

[1995] Specific example

[1996] For example, in a sales meeting, by inputting the following prompt into the AI ​​model, it is possible to understand the specific steps for creating preliminary materials.

[1997] Example of a prompt:

[1998] For next week's sales meeting, please automatically generate a PowerPoint presentation containing sales performance data for the past six months and information on new products.

[1999] This prompt indicates how the process will proceed. The server will use the Google Calendar API to retrieve the schedule, search for data in SQL Server and MongoDB, and generate a document using python-pptx. Finally, it will use the SharePoint API to save the document and notify the user.

[2000] The flow of the specific processing in Example 1 will be explained using Figure 11.

[2001] System program processing flow

[2002] Implementation Example of Preliminary Document Preparation

[2003] Step 1:

[2004] The server uses the Google Calendar API to retrieve the company's meeting schedule for the following day. This involves the server sending a request to the API and receiving schedule data in JSON format as a response. For example, it might confirm that a "sales meeting" is scheduled for 2:00 PM the following day.

[2005] Input: Request to the Google Calendar API

[2006] Output: Schedule data in JSON format

[2007] Specific operation: Parse JSON data and extract meeting type, date and time, and participant information.

[2008] Step 2:

[2009] Based on the acquired schedule data, the server searches for relevant data such as past sales performance and product information from SQL Server and MongoDB. This allows the server to execute SQL queries to retrieve the necessary performance data and MongoDB queries to retrieve product information.

[2010] Input: Schedule data

[2011] Output: Search results from the performance database and information database.

[2012] Specific operation: Issue specific queries to SQL Server and MongoDB and collect relevant data.

[2013] Step 3:

[2014] The server uses libraries such as "python-pptx" and "ReportLab" to automatically generate preliminary materials in PowerPoint and PDF formats from the integrated data. The server displays past sales performance as graphs and places new product information as tables and text on the slides.

[2015] Input: Performance data and product information

[2016] Output: Preliminary materials in PowerPoint or PDF format

[2017] Specific actions: Format the data and generate slides according to the specified template.

[2018] Step 4:

[2019] The server saves the generated documents to the company's document management system using the SharePoint API. The server then sends a completion notification to the user.

[2020] Input: Generated preliminary data

[2021] Output: URL of documents stored in SharePoint and user notification

[2022] Specific actions: Upload documents to SharePoint and send email notifications via an SMTP server.

[2023] Automatic meeting minutes creation system

[2024] Step 1:

[2025] The terminal records audio data during the meeting in real time and sends it to the server. The terminal captures the audio via a "sound card" and streams the audio data to the server using WebSocket.

[2026] Input: Real-time audio data

[2027] Output: Audio data sent to the server

[2028] Specific operation: Captures audio input from the microphone and transfers the data via a WebSocket connection.

[2029] Step 2:

[2030] The server uses the Google Cloud Speech-to-Text API to convert received audio data into text. It sends audio data to the API and receives text data as a response.

[2031] Input: Audio data

[2032] Output: Text data

[2033] Specific operation: Divide the audio data into fixed batches, send them to the API, and retrieve the conversion results.

[2034] Step 3:

[2035] The server uses "SpaCy" to analyze text data and classify it by speaker. It extracts speaker names and topics from the text data and categorizes them accordingly.

[2036] Input: Text data

[2037] Output: Classified text data

[2038] Specific operation: Performs text analysis, extracts and classifies speaker names and keywords.

[2039] Step 4:

[2040] The server automatically generates meeting minutes based on text data categorized using the Google Docs API. The generated meeting minutes are saved to Google Drive.

[2041] Input: Classified text data

[2042] Output: Meeting minutes saved to Google Drive

[2043] Specific actions: Insert data into a template, generate a document, and save it.

[2044] Step 5:

[2045] The server will notify the user of the URL of the generated meeting minutes. Notification methods include email and in-app notifications.

[2046] Input: URL of the generated meeting minutes

[2047] Output: Completion notification to the user

[2048] Specific operation: Use an SMTP server to send email notifications and generate in-app notifications.

[2049] Automatic question and answer processing system

[2050] Step 1:

[2051] Users can type or speak questions during the meeting. The device records the questions and, if spoken, converts them to text using the Google Cloud Speech-to-Text API.

[2052] Input: Question in voice or text format

[2053] Output: Text version of the question content

[2054] Specific operation: The user either types their speech or captures the audio and converts it to text.

[2055] Step 2:

[2056] Based on the transcribed query, the server uses "ElasticSearch" to retrieve relevant information from the company's internal portal and databases.

[2057] Input: Text-based question content

[2058] Output: Search Results

[2059] Specific operation: Send a query to Elasticsearch and retrieve relevant information.

[2060] Step 3:

[2061] Based on the acquired information, the server automatically generates answers using a generative AI model (e.g., "GPT-3").

[2062] Input: Search Results

[2063] Output: Auto-generated answer

[2064] Specific operation: Input prompts into the generative AI model and generate responses.

[2065] Step 4:

[2066] The server uses the Gmail API to generate the response, then creates and saves an email draft.

[2067] Input: Auto-generated answer

[2068] Output: Email draft

[2069] Specific operation: Create and save an email draft using the Gmail API.

[2070] Step 5:

[2071] The user is notified of the completed email draft. The user reviews the draft, makes any necessary revisions, and sends the email.

[2072] Input: Email draft

[2073] Output: Completion notification and correction / sent email to the user.

[2074] Specific operation: An SMTP server is used to send a notification, and the user sends an email after making corrections.

[2075] (Application Example 1)

[2076] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2077] In modern factories, meetings and work reports cover a wide range of topics, requiring significant time and effort for preparation and execution. In particular, creating preliminary materials, preparing meeting minutes, and responding quickly to questions are laborious tasks. This situation leads to decreased meeting efficiency and hinders productivity improvements. A system is needed to address these challenges and streamline meetings and work reports within factories.

[2078] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[2079] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a performance database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for saving the generated document and notifying the user, means for recording audio data during a meeting in real time and sending it to the server, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for automatically generating meeting minutes based on the classified text data, means for saving the generated meeting minutes and notifying the user, means for converting the content of questions entered during the meeting into text and searching the company portal and database, means for automatically generating answers based on the search results, means for creating an email draft based on the generated answers, means for saving the created email draft and notifying the user, and means for generating answers based on prompt sentences using a generation AI model. As a result, the creation of pre-meeting materials, the creation of meeting minutes, and the processing of questions and answers are automated, enabling efficient meeting progress and rapid decision-making.

[2080] "Schedule data" refers to data that shows the planned dates for meetings and tasks.

[2081] A "performance database" is a database that stores records of past work and meetings.

[2082] An "information database" is a database that stores various kinds of related information.

[2083] "Specified format" refers to a standard that specifically instructs the format and layout of a document.

[2084] A "document" refers to a digital document or paper-based material that organizes information according to a specific format.

[2085] "Audio data" refers to data that includes audio waveform data and its digitized form.

[2086] "Text data" refers to data obtained by converting audio data into written text.

[2087] "Speakers" refer to individuals or groups who make statements during meetings or discussions.

[2088] "Meeting minutes" are documents that record the content of meetings or discussions.

[2089] "Questions" refers to the content of questions submitted during meetings or discussions.

[2090] An "internal portal" is an information sharing platform used within a company.

[2091] A "database" is a system that systematically stores and manages data in a searchable format.

[2092] A "draft email" is a draft of an email document before it is sent.

[2093] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to perform data processing or document generation.

[2094] A "prompt sentence" is a sentence containing instructions or questions that are input into a generative AI model.

[2095] This invention is a system aimed at improving the efficiency of meetings and work reports within a factory, and a specific embodiment thereof is shown below. This system includes a server, terminals, and user operation.

[2096] server

[2097] The server uses the following hardware and software:

[2098] Hardware: High-performance processor, ample memory, SSD storage

[2099] Software: Schedule management system, performance database, information database, natural language processing engine, speech recognition engine

[2100] The server first accesses the company's schedule management system and retrieves schedule data. Based on the retrieved schedule data, it searches for relevant information from the performance database and information database, integrates this information, and automatically generates preliminary materials in the specified format (e.g., PDF or PowerPoint). These generated materials are stored in the company's document management system and notified to the user.

[2101] Furthermore, the server receives audio data transmitted from terminals during the meeting in real time and converts it into text using a speech recognition engine. The converted text data is analyzed by a natural language processing engine and classified by speaker. Based on the classified text data, meeting minutes are automatically generated, saved, and then notified to the user.

[2102] Furthermore, the system transcribes questions entered during meetings into text, searches the company portal and databases to retrieve relevant information, and automatically generates answers based on that information, creating a draft email. This draft email is saved and the user is notified. The system also has a function where the generation AI model generates answers based on prompt text.

[2103] terminal

[2104] The devices are smartphones, tablets, or factory robots used within the factory. They are used as follows:

[2105] Collection of audio data

[2106] Record of questions entered during the meeting

[2107] Receipt and confirmation of notification

[2108] The terminal records audio data from the meeting in real time and sends it to the server. Users can also use the terminal to input questions during the meeting. The terminal receives notifications and informs users when materials or meeting minutes are complete.

[2109] User

[2110] The user performs the following actions:

[2111] Receive notifications from the system

[2112] Review the prepared preliminary documents and make any necessary corrections.

[2113] Review the automatically generated meeting minutes and make corrections if necessary.

[2114] Review the draft email, make any necessary corrections, and then send it.

[2115] Specific example

[2116] As a concrete example, the following prompt statements are used as input to the generating AI model:

[2117] Prompt: Based on the schedule data and past performance data, please prepare the meeting materials for the following day.

[2118] data:

[2119] Schedule: Next day's schedule: Safety meeting

[2120] Track record: Past safety reports

[2121] In this way, the creation of preliminary materials, the preparation of meeting minutes, and the processing of questions and answers are automated, significantly improving the efficiency of meetings and work reports within the factory.

[2122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[2123] Step 1:

[2124] The server accesses the company's schedule management system and retrieves schedule data. This retrieved schedule data becomes the input for the current process. Based on this data, the server proceeds to the next processing step.

[2125] Step 2:

[2126] The server searches for relevant information from the performance database and information database based on the acquired schedule data. The results of the search, including performance data and related information, are output. This output is used to create preliminary documents.

[2127] Step 3:

[2128] The server integrates the performance data and related information obtained in Step 2 and automatically generates a document in the specified format. Specifically, it outputs the document as a PDF or PowerPoint file. This output document is used as preliminary material.

[2129] Step 4:

[2130] The server saves the generated document to the company's document management system and sends a notification to the user when saving is complete. The user receives the notification and downloads and reviews the document as needed.

[2131] Step 5:

[2132] The terminal records audio data during the meeting in real time and sends that data to the server. The audio data arrives at the server as input and proceeds to the next processing step.

[2133] Step 6:

[2134] The server converts the transmitted audio data into text data using a speech recognition engine. This converted text data is then output and analyzed.

[2135] Step 7:

[2136] The server analyzes the converted text data using a natural language processing engine and classifies it by speaker. Text data is used as input, and the classified text data is obtained as output.

[2137] Step 8:

[2138] The server automatically generates meeting minutes based on the categorized text data. The generated minutes are output in a specific format and stored in the company's document management system.

[2139] Step 9:

[2140] The server sends a notification to the user when the generated meeting minutes have been saved. The user receives the notification and reviews the minutes, making any necessary corrections.

[2141] Step 10:

[2142] The system transcribes questions entered by users on their devices during meetings into text and sends this text data to a server. The server analyzes this text data and searches the company portal and databases to retrieve relevant information.

[2143] Step 11:

[2144] The server automatically generates a response using a generative AI model based on the acquired relevant information. The prompt text and relevant information are used as input, and text data is output as the response.

[2145] Step 12:

[2146] The server creates an email draft based on the generated response, saves this draft, and notifies the user. The user receives the notification, reviews the email content as needed, makes any necessary corrections, and then sends it.

[2147] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2148] Pre-document creation system

[2149] This invention is a system for improving the efficiency of corporate meetings, and is characterized by its ability to generate more personalized materials, meeting minutes, and Q&A responses by taking user emotions into consideration. This system provides automatic creation of pre-meeting materials, automatic creation of meeting minutes, efficient processing of questions and answers, and adjustments by an emotion engine.

[2150] Implementation Example of Preliminary Document Preparation

[2151] 1. Obtaining schedule data

[2152] The server accesses the company's schedule management system to check the meeting schedule for the next day. For example, it might find that there is a meeting named "Sales Meeting".

[2153] 2. Searching for related data

[2154] The server uses the schedule data to search for relevant information from the performance database and information database. For example, it retrieves past sales performance and product information.

[2155] 3. Emotion recognition by an emotion engine

[2156] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[2157] 4. Automatic document generation and adjustment

[2158] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[2159] 5. Notifications and saving

[2160] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[2161] Automatic meeting minutes creation system

[2162] 1. Collection of audio data

[2163] The terminal records audio data during the meeting in real time and streams it to the server.

[2164] 2. Speech Recognition and Text Conversion

[2165] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[2166] 3. Text Classification and Analysis

[2167] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[2168] 4. Analysis using an emotional engine

[2169] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[2170] 5. Automatic generation and adjustment of meeting minutes

[2171] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[2172] 6. Saving and Notifications

[2173] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[2174] Automatic question and answer processing system

[2175] 1. Question collection and text transcription

[2176] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[2177] 2. Text conversion using speech recognition

[2178] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[2179] 3. Searching for related information

[2180] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[2181] 4. Feedback from the Emotion Engine

[2182] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[2183] 5. Automatic response generation and email draft creation

[2184] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[2185] 6. Notification and Confirmation

[2186] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[2187] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, and provide advanced personalization that takes user emotions into account, thereby reducing the burden on employees and improving productivity.

[2188] The following describes the processing flow.

[2189] Processing of the program for creating preliminary documents

[2190] Step 1:

[2191] The server accesses the company's schedule management system to check the meeting schedule for the following day.

[2192] Step 2:

[2193] The server extracts information such as the type of meeting, start time, and attendees from the schedule data. For example, it might retrieve that there is a meeting named "Sales Meeting".

[2194] Step 3:

[2195] Based on the schedule data, the server accesses the performance database and retrieves relevant sales performance data. This includes past monthly reports and quarterly results.

[2196] Step 4:

[2197] The server accesses the company database to retrieve information on new products and the latest company news. This information will be relevant to the agenda of the meeting.

[2198] Step 5:

[2199] The server uses an emotion engine to recognize the emotional data the user has previously displayed in meetings. For example, it analyzes whether the user showed stress or satisfaction in past meetings.

[2200] Step 6:

[2201] The server integrates acquired performance data and company profile data, and automatically generates preliminary materials in specified formats such as PowerPoint and PDF. Furthermore, it adjusts the tone and content of the materials based on data from the emotion engine. For example, it increases positive expressions to reduce stress.

[2202] Step 7:

[2203] The server uploads the generated preliminary documents to the company's cloud storage and sends a completion notification to the user. The user checks the notification, downloads the documents, and uses them.

[2204] Processing of the program for automatically creating meeting minutes

[2205] Step 1:

[2206] The terminal records audio data during the meeting in real time and streams the data to the server.

[2207] Step 2:

[2208] The server receives the audio data and uses a speech recognition algorithm to convert the speech into text. For example, the audio "It's time for the sales report" is converted to text "It's time for the sales report".

[2209] Step 3:

[2210] The server analyzes the text data and classifies it based on factors such as the speaker's identity, the content of the statement, and the topic. For example, it might classify the speaker as "Sales Manager" and the content as "Sales Report."

[2211] Step 4:

[2212] The server uses an emotion engine to analyze the emotional data of speakers during a meeting. For example, it can assess whether they are showing anxiety or excitement regarding a particular topic.

[2213] Step 5:

[2214] The server automatically generates meeting minutes in a pre-configured format based on classified text data and sentiment data. Based on the sentiment data, the minutes are adjusted to draw attention to specific sections.

[2215] Step 6:

[2216] The server saves the generated meeting minutes to the company's database and sends a completion notification to the user. The user receives the completion notification for the meeting minutes and reviews its contents.

[2217] Program processing for automated question and answer session

[2218] Step 1:

[2219] Users can type questions as text or speak them aloud during the meeting. The device records these questions.

[2220] Step 2:

[2221] When a question is received via voice, the server uses speech recognition technology to convert the question into text. For example, if the voice says, "What are next month's sales targets?", it will be converted to text as "What are next month's sales targets?".

[2222] Step 3:

[2223] The server searches the internal portal and corporate database based on the text-based questions and retrieves relevant information.

[2224] Step 4:

[2225] The server uses an emotion engine to analyze the questioner's emotional data. For example, it evaluates whether the questioner is showing dissatisfaction or interest during the questioning process.

[2226] Step 5:

[2227] The server automatically generates responses based on the information it receives, adjusting the tone and content of the emails based on sentiment data. For example, if the user expresses dissatisfaction, it will generate a more polite response.

[2228] Step 6:

[2229] The server saves the generated email draft to the company's mail server and sends a notification to the user that the draft is complete. The user receives the notification, reviews the email draft, and makes any necessary revisions.

[2230] Through the above processing steps, this system can automate meeting preparation, recording, and question handling, provide advanced personalization that takes user emotions into account, reduce employee workload, and improve productivity.

[2231] (Example 2)

[2232] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2233] Traditional meeting management systems often require manual processes for meeting preparation, minute-taking, and Q&A, resulting in significant time and effort. Furthermore, these tasks often fail to consider user emotions, impacting meeting efficiency and participant satisfaction. Therefore, there is a need for a system that automates meeting preparation, recording, and Q&A, while also providing personalized responses that take user emotions into account.

[2234] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2235] In this invention, the server includes means for acquiring schedule data, means for searching for relevant information from a history database and an information database based on the acquired schedule data, means for integrating the searched information and automatically generating a document in a specified format, means for adjusting the representation of the data using an emotion analysis engine when generating the document, means for saving the generated document and notifying the user, means for transmitting audio data to the server in real time, means for converting the transmitted audio data into text, means for analyzing the text data and classifying it by speaker, means for analyzing emotions based on the text data, means for automatically generating meeting minutes based on the classified and emotion-analyzed text data, means for searching an internal portal and a database based on the transcribed question content, means for automatically generating answers based on the search results, means for adjusting the representation of the answers using an emotion analysis engine when generating the answers, means for creating an email draft based on the generated answers, and means for saving the created email draft and notifying the user. This automates meeting preparation, recording, and question handling, and enables advanced personalization that takes user emotions into consideration.

[2236] "Schedule data" refers to data that includes a user's schedule and appointments. It is obtained from the company's schedule management system.

[2237] A "history database" refers to a database that stores data on past achievements and activities.

[2238] An "information database" refers to a database that stores data such as product information and company policies.

[2239] An "emotion analysis engine" refers to software or a system that analyzes a user's emotions from text or audio and outputs the results as emotion data.

[2240] "Documents" refer to digital files containing organized and edited information such as meeting materials, minutes, and email drafts.

[2241] "Specified format" refers to the predetermined format and layout used when creating a document.

[2242] "Audio data" refers to data that digitally saves the content of participants' remarks recorded during a meeting.

[2243] "Text" refers to data obtained by converting audio data into written information.

[2244] "Classification" refers to dividing speakers and their statements into specific categories based on text data.

[2245] "Meeting minutes" refers to a document that records the content and statements made during a meeting.

[2246] An "internal portal" refers to a web-based platform used for information sharing and communication within a company.

[2247] "Answer" refers to the information provided as a response to a question.

[2248] A "draft email" refers to the content of an email that has been saved in its state before being sent to the user.

[2249] This invention relates to a system that automates meeting preparation, minute-taking, and question-and-answer processing, and further generates more personalized materials, minutes, and Q&A by taking user emotions into consideration. This system mainly consists of a server, terminals, and users, and is implemented using the following specific hardware and software.

[2250] Pre-document creation system

[2251] 1. Hardware and software configuration

[2252] The server accesses a schedule management system (e.g., Google Calendar) to retrieve data. Python is the primary programming language, and MySQL and PostgreSQL are used as databases.

[2253] The server uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's emotions and use that information to adjust the content of the materials.

[2254] The server uses a combination of Python libraries to automatically generate pre-recorded materials in PowerPoint or PDF format (e.g., python-pptx, ReportLab).

[2255] 2. Specific Examples

[2256] The server retrieves the schedule data for the next day's "sales meeting" and searches the database for past sales performance data and the latest product information. Using an emotion analysis engine, it automatically generates pre-meeting materials that heavily utilize positive language, based on data showing that the user exhibited high stress levels in past sales meetings.

[2257] Example prompts for generative AI models

[2258] Please prepare the preliminary materials for the next day's "Sales Meeting" using the following information and maintaining a positive tone.

[2259] September sales: ¥10,000,000

[2260] New product XYZ specification update

[2261] To reduce the user's past stress

[2262] Automatic meeting minutes creation system

[2263] 1. Hardware and software configuration

[2264] The terminal uses its microphone to record audio data during the meeting and utilizes WebSocket to stream it to the server in real time.

[2265] The server uses the Google Speech-to-Text API to convert the audio data into text.

[2266] The server uses natural language processing tools (e.g., NLTK) to analyze text data and classify it by speaker. It also uses a sentiment analysis engine to analyze emotions.

[2267] 2. Specific Examples

[2268] The terminal records the audio during the meeting and sends it to the server in real time. The server converts the audio, such as "It's time for the sales report," into text, classifies the speaker and the content of their statement, and also performs sentiment analysis.

[2269] Example prompts for generative AI models

[2270] Please create meeting minutes that include the following audio data. Please pay attention to the emotions of the speakers.

[2271] Sales Manager: "It's time for the sales report."

[2272] Emotional data: "Anxiety"

[2273] Automatic question and answer processing system

[2274] 1. Hardware and software configuration

[2275] Users can type questions into their devices or speak them aloud during the meeting. The devices record these questions and send them to the server.

[2276] The server uses the Google Speech-to-Text API to convert voice questions into text and then uses SQL queries to search for relevant information in the company's database.

[2277] The server uses an emotion analysis engine to analyze the questioner's emotions and adjusts the tone of the response accordingly.

[2278] 2. Specific Examples

[2279] If a user asks "What are next month's sales targets?" during a meeting, the device records the question and sends it to the server. The server converts the audio to text, searches for relevant information, and generates a polite response based on sentiment analysis.

[2280] Example prompts for generative AI models

[2281] Please generate answers to the following questions in a polite tone.

[2282] Question text: "What are your sales targets for next month?"

[2283] Emotional data: "Dissatisfaction"

[2284] Through the system configuration and processing steps described above, this invention automates meeting preparation, recording, and question handling, while also providing a high level of personalization that takes user emotions into consideration. This is expected to reduce the burden on employees and improve productivity.

[2285] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2286] Pre-document creation system

[2287] Step 1:

[2288] The server sends an HTTP request to the company's scheduling management system (e.g., Google Calendar) and retrieves the next day's meeting schedule data in JSON format.

[2289] input:

[2290] API endpoint for the schedule management system

[2291] HTTP Request

[2292] output:

[2293] Retrieved schedule data (JSON format)

[2294] Specific actions:

[2295] The server sends an HTTP request and receives schedule data in JSON format. For example, it retrieves data for "October 11, 2023, 9:00 - 10:00 Sales Meeting".

[2296] Step 2:

[2297] Based on the retrieved schedule data, the server issues SQL queries to search for the necessary data from the history database (e.g., MySQL) and the information database (e.g., PostgreSQL).

[2298] input:

[2299] Acquired schedule data

[2300] SQL query

[2301] output:

[2302] Search results include historical data and product information.

[2303] Specific actions:

[2304] Based on the acquired schedule data, the server issues an SQL query to retrieve historical data such as "Sales in September 2023: ¥10,000,000" and product information such as "Specification update f...

Claims

1. Means of obtaining schedule data, A means for searching for relevant information from the performance database and information database based on the acquired schedule data, A means of integrating searched information and automatically generating a document in a specified format, A means of saving the generated document and notifying the user, A system that includes this.

2. A means of sending audio data to a server in real time, A means of converting transmitted audio data into text, A method for analyzing text data and classifying it by speaker, A method for automatically generating meeting minutes based on classified text data, A means of saving the generated meeting minutes and notifying the user, The system according to claim 1, including the following:

3. A means of searching the company portal and database based on the textual content of the questions, A means of automatically generating answers based on search results, A method for creating an email draft based on the generated response, A means to save the created email draft and notify the user, The system according to claim 1, including the following:

Images