system

Abstract

We provide the system. [Solution] A sound recognition means that recognizes speech in real time and converts it into language data, An information extraction method that uses natural language processing to extract important features from language data and automatically generates records, A record distribution means for distributing the generated records to communication devices, A task management system that automatically lists and manages tasks that arise during a meeting, A scheduling tool that adjusts meeting schedules based on date information and provides necessary information immediately, An information aggregation means that quickly gathers on-site information in situations requiring action, extracts important keywords, and automatically generates reports. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a meeting, in order to enable participants to focus only on discussions, it is necessary to efficiently handle miscellaneous tasks such as creating minutes, extracting important information, organizing tasks, and progress management. However, these tasks require time and effort and contribute to reducing the efficiency of the meeting. In addition, there is a problem that immediate provision of information related to the meeting and visual and easy-to-understand management of tasks are required, but are insufficient in many current systems.

Means for Solving the Problems

[0005] This invention provides a speech recognition means that recognizes audio during a meeting in real time and converts it into text data. It also includes a key point extraction means that extracts important information from the converted text data using natural language processing and automatically generates meeting minutes. Furthermore, it includes a task management means that quickly distributes the generated meeting minutes and lists and manages meeting tasks. This realizes a system that provides an environment in which participants can concentrate on the discussion, immediately obtain the information necessary for the meeting, and visually check the progress of tasks.

[0006] "Speech recognition means" refers to devices or software that have the function of converting audio from a meeting into digital text data in real time.

[0007] A "key point extraction tool" is a device or software that uses natural language processing technology to select important information and topics from text data and compile them into meeting minutes.

[0008] A "meeting minutes distribution method" refers to a device or software that has the function of quickly transmitting generated meeting minutes to the terminals of relevant parties via a network.

[0009] A "task management tool" is a device or software that has the function of automatically identifying, listing, and efficiently managing tasks that arise during a meeting.

[0010] A "scheduling adjustment tool" refers to a device or software that has the function of scheduling meetings based on date information and providing related information.

[0011] "Means of providing materials" refers to devices or software that have the function of searching for and presenting relevant materials in real time during and after a meeting.

[0012] "Progress management tools" are devices or software that have the function of managing task deadlines and assignees, and visually tracking their progress. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that improves the efficiency of meetings by integrating speech recognition technology and natural language processing technology. Users start a meeting using a meeting terminal. The terminal captures the audio during the meeting in real time via a microphone and transmits the audio data to a server.

[0035] The server quickly converts audio data into text data using a speech recognition module. This converted text data is then processed by a natural language processing module on the server to extract key points and generate an outline of the meeting minutes. The meeting minutes are automatically generated by the server and immediately delivered to the terminal. Users can review the minutes on their terminal and add comments as needed.

[0036] Furthermore, the server automatically lists any new tasks that arise during the meeting. Each task is accompanied by information such as a deadline and the person responsible, and is managed within the server. Users can check this task information from their terminals and manage its progress.

[0037] Furthermore, the server adjusts meeting schedules based on the user's schedule information. It also instantly searches for relevant documents and data as needed and provides them to the user's terminal. This ensures that users can access information without delay during meetings and focus on the discussion.

[0038] As a concrete example, consider a user holding a monthly reporting meeting. The user's device connects to the server at the start of the meeting and transmits what is said during the meeting to the server in real time. The server transcribes the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion about cost reduction takes place, the content is summarized and immediately shared with all participants.

[0039] This invention significantly improves meeting productivity and provides a system that helps users concentrate on the discussion.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The terminal captures audio via its microphone at the start of the meeting. The captured audio data is divided into packets in real time and sent to the server.

[0043] Step 2:

[0044] The server inputs the received audio packets into the speech recognition module, converting the audio into text data. This allows the audio information to be treated as text information.

[0045] Step 3:

[0046] The server sends the text data to a natural language processing module to extract important information and key points from the meeting. The extracted information is temporarily stored on the server.

[0047] Step 4:

[0048] The server automatically creates meeting minutes based on the extracted key points. The created meeting minutes data is then sent back to the terminal.

[0049] Step 5:

[0050] Users can review meeting minutes received on their devices and add comments or corrections as needed. The minutes can also be saved and shared with other participants.

[0051] Step 6:

[0052] The server automatically creates a list of tasks discussed during the meeting and adds deadlines and assignees to each task. The generated task list is then delivered to the terminals.

[0053] Step 7:

[0054] Users can view their task list on their device and manage their progress. The completion status of each task is synchronized with the server, making it easy to track overall progress.

[0055] Step 8:

[0056] The server integrates with the user's calendar data to schedule the next meeting. It also displays necessary documents and information on the user's device during the meeting, improving meeting efficiency.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] Modern meetings are becoming increasingly complex, and with the growing volume and processing speed of information, efficient information management and record-keeping are required to allow participants to focus on the discussion. However, manual creation of meeting minutes, task management, and provision of related materials are time-consuming and labor-intensive, potentially leading to decreased productivity. Therefore, there is a need for a means to properly manage meeting information in real time and to efficiently advance discussions.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes speech recognition means, key point extraction means, and meeting minute distribution means. This makes it possible to convert audio during a meeting into text data in real time, automatically extract key points of important discussions to generate meeting minutes, and immediately distribute them to participants. This promotes the immediate sharing of information and helps participants concentrate on the discussion.

[0062] "Speech recognition means" refers to a technology or device for converting speech data into text data in real time.

[0063] A "key point extraction method" is a technique or device that uses natural language processing to extract important information or key points of discussion from text data.

[0064] "Meeting minutes distribution means" refers to a technology or device that distributes generated meeting minutes to information terminals via communication.

[0065] "Work item management means" refers to a technology or device for automatically listing work items that arise during a meeting and for managing those work items.

[0066] "Schedule adjustment means" refers to technology or equipment for adjusting meeting schedules based on time information and providing necessary information immediately.

[0067] "Means of providing information" refers to a technology or device that provides information related to an ongoing meeting in real time based on text data acquired by speech recognition means.

[0068] "Progress management means" refers to a technology or device for identifying deadlines and responsible persons for work items extracted by work item management means, and for individually managing their progress.

[0069] This invention is a system that combines speech recognition technology and natural language processing technology, designed to streamline meetings. Users initiate a meeting using a conference terminal. The terminal uses a high-sensitivity microphone to capture audio in real time during the meeting. The audio data is stored digitally and transmitted to a server via a secure protocol (e.g., TLS / SSL).

[0070] The server uses a speech recognition module (for example, Google® Speech-to-Text API, a common speech recognition API) to quickly convert audio data into text data. Then, a natural language processing module (e.g., SpaCy or NLTK) is used to analyze the converted text data and extract the key points of the meeting. The automatically generated meeting minutes are then immediately delivered to terminals by the server.

[0071] Users can view meeting minutes on their devices and add comments as needed. In this process, the server has a function to list new tasks that arise during the meeting. These tasks are accompanied by deadlines and assigned personnel information generated by an AI model, and are viewed and managed through the device. Furthermore, the server adjusts meeting schedules based on the user's time information and instantly searches for and provides necessary materials and data. Therefore, users can access necessary information without delay, even during meetings, enabling efficient discussions.

[0072] As a concrete example, consider a scenario where a user holds a regular progress report meeting. In this case, the user's device connects to the server at the start of the meeting and transmits audio to the server in real time. The server converts the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion takes place regarding the efficiency of management resources, the content is summarized and immediately shared with all participants.

[0073] An example of a prompt message would be, "Please extract the key points from the following audio and create meeting minutes." This would significantly improve meeting productivity and provide users with an environment where they can focus on the discussion.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] Users initiate a meeting using a conference terminal. The terminal captures the meeting audio in real time using its built-in high-sensitivity microphone. The input is the audio from the meeting, and the output is a digital audio file (e.g., WAV format). The terminal temporarily saves the audio data to a PC or online storage to prevent data loss.

[0077] Step 2:

[0078] The terminal sends the stored audio data to the server using a secure protocol (e.g., TLS / SSL). This process involves data encryption. The input is the captured audio data, and the output is the encrypted audio data. This ensures secure data transfer.

[0079] Step 3:

[0080] The server converts received audio data into text data using a speech recognition module (e.g., a common speech recognition API). The input is the received audio data, and the output is text data (e.g., in text format). The server analyzes the audio waveform data, recognizes phonemes and words, and performs this conversion.

[0081] Step 4:

[0082] The server analyzes the generated text data using a natural language processing module (e.g., SpaCy) to extract the key points of the meeting. The input is the converted text data, and the output is a summary of the key points (e.g., summary text). The server calculates keyword frequency and sentence weighting within the text to extract important information.

[0083] Step 5:

[0084] The server automatically generates meeting minutes based on extracted key points. The input is data summarizing the key points, and the output is a formatted meeting minute (e.g., PDF or DOC format). This generation process utilizes a template engine and style processing. The server then distributes the generated meeting minutes to information terminals.

[0085] Step 6:

[0086] The server automatically lists work items that arise during a meeting using a generation AI model. The input is text data of the meeting content, and the output is a list of work items. The server analyzes the command structure and task-related expressions within the text to identify the items.

[0087] Step 7:

[0088] Users can review meeting minutes and work items delivered via their terminals and add comments and corrections. Inputs are the delivered meeting minutes and work items, while outputs are supplementary meeting minutes and an updated work item list. Users can manage their progress through their terminals.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In emergency situations, there is a problem in quickly and accurately gathering on-site audio information and transmitting it to the command center and response team with speed and accuracy. In particular, it is necessary to efficiently assess the importance of information generated on-site, quickly extract the necessary information, and share it.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes an acoustic recognition means, an information extraction means, and an information aggregation means. This makes it possible to quickly convert voice information into text, automatically extract important information, and immediately provide a report to the command center at the scene of an emergency response.

[0094] "Acoustic recognition means" refers to a means of acquiring sound as a digital signal and converting it into language data in real time.

[0095] An "information extraction method" is a means of extracting important features and keywords from acquired language data using natural language processing and automatically generating records.

[0096] A "record distribution method" is a means of distributing generated records to communication devices and sharing them with relevant parties.

[0097] A "task management system" is a means of automatically listing and managing tasks that arise during meetings or interactions.

[0098] A "schedule adjustment method" is a means of adjusting meeting schedules based on date information and providing necessary information immediately.

[0099] An "information aggregation method" is a means of aggregating on-site audio information, extracting important keywords, and generating an automated report in situations requiring action.

[0100] One embodiment of this invention consists of a system integrating acoustic recognition means, information extraction means, recording and distribution means, and the like. This system uses specific hardware and software to acquire sound in real time and convert it into language data.

[0101] Specifically, a device such as a smartphone captures audio data with its microphone and converts it into language data using the Google Cloud Speech-to-Text API. This speech recognition requires an acoustic model using TENSORFLOW®.

[0102] The converted language data is sent to a server, where the SpaCy natural language processing library is used by an information extraction system to identify key keywords and automatically generate records. The generated records are immediately distributed to relevant parties using cloud communication capabilities. The information aggregation system quickly processes information that is particularly needed in emergencies and provides important content to command centers and other relevant locations.

[0103] In the operation of this system, the server lists tasks as they arise and provides a task management function that makes it easy for users to check their progress from their terminals. In addition, the scheduling mechanism allows for quick and efficient management of meeting dates, minimizing inconvenience caused by schedule changes.

[0104] For example, if keywords requiring emergency response, such as "fire," "smoke," and "evacuation," are identified at a fire response site, a report containing these keywords will be immediately generated and distributed. This enables rapid information sharing between the field and the command center.

[0105] An example of a prompt for the generating AI model would be the text: "Please tell me how to extract important keywords from audio data at an emergency response site and automatically generate a report to the command center immediately."

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The device uses a microphone to capture audio data from the field and generates a digitized audio signal. This signal is sent to the Google Cloud Speech-to-Text API. The input is analog audio, and the output is a digitized audio signal.

[0109] Step 2:

[0110] The server uses the Google Cloud Speech-to-Text API to analyze the speech signal and convert it into language data in real time. An acoustic model is used for this conversion, sequentially outputting text data from the speech data. The input is a digitized speech signal, and the output is text data.

[0111] Step 3:

[0112] The server uses SpaCy to extract important keywords from text data. Natural language processing is employed here to clarify key features from the linguistic data. The input is text data, and the output consists of extracted keywords and important content.

[0113] Step 4:

[0114] As a means of information aggregation, the server instantly generates important reports based on extracted keywords. These reports contain all the necessary content and are quickly transmitted to the command center. The input is the important information, and the output is the generated report.

[0115] Step 5:

[0116] Users view reports generated on their devices and report the situation to the command center as needed. User review of the reports facilitates smoother communication. The input is the generated reports, and the output is the user's actions and feedback.

[0117] Through this process, we will utilize generative AI models to enable immediate processing and transmission of information at emergency sites.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] This invention is a system that improves meeting efficiency and participant engagement by incorporating an emotion engine that recognizes user emotions, in addition to speech recognition technology and natural language processing technology. The system is composed of three main elements: a server, terminals, and users.

[0120] When a user starts a meeting, their device uses its microphone and camera to capture audio and video data in real time. The device sends the captured data to a server. The server uses a speech recognition module to convert the audio into text data, and simultaneously analyzes the user's facial expressions and tone of voice from the video data to recognize their emotions.

[0121] The server uses a natural language processing module to extract important information from text data, summarize the key points of the meeting, and generate meeting minutes. Simultaneously, an emotion engine identifies the user's emotional state, and the meeting's progress can be dynamically adjusted based on this information. For example, if a participant is deemed confused, an alert will be displayed on the terminal to provide additional explanations or relevant materials.

[0122] Users can view meeting minutes generated on their devices, and tasks that arise during the meeting are displayed in real time. The server assists in optimizing the meeting's outcome based on user sentiment information. This creates an environment where discussions are not one-sided and all participants can actively engage.

[0123] As a concrete example, when holding a strategy meeting, the user activates a terminal to start the meeting. The terminal captures audio and video, and the server performs analysis. If a presentation is given during the meeting and the emotion engine detects a decline in participants' interest or understanding, the server suggests options to readjust the focus of the meeting based on that information. In this way, this invention functions not only as a means of transmitting information, but also as a tool to improve the quality of communication.

[0124] The following describes the processing flow.

[0125] Step 1:

[0126] The device captures audio and video using its built-in microphone and camera as soon as the meeting starts. The captured data is sent to the server in real time.

[0127] Step 2:

[0128] The server converts the audio data received by the speech recognition module into text data. This process transcribes the audio content into text.

[0129] Step 3:

[0130] The server simultaneously analyzes the video data using an emotion engine, recognizing the user's emotional state from their facial expressions, facial movements, and voice tone. Changes in emotion and specific states are identified.

[0131] Step 4:

[0132] The server analyzes text data through a natural language processing module and extracts important information and key points from the meeting. Based on the extraction results, meeting minutes are automatically generated.

[0133] Step 5:

[0134] Based on the emotional information detected by the emotion engine, the server analyzes the user's level of understanding and reactions, and provides suggestions for meeting progress or additional materials if necessary. This information is then notified to the terminal.

[0135] Step 6:

[0136] Users can review the meeting minutes and suggested supplementary information generated on their devices, allowing them to gain a deeper understanding of the meeting content and participate in the discussion.

[0137] Step 7:

[0138] The server automatically lists any new tasks that arise during the meeting, identifying deadlines and assigning responsibilities. The generated task list is also delivered to the user's terminal, allowing them to monitor progress.

[0139] Step 8:

[0140] Based on user feedback, the server prepares post-meeting feedback and provides it to users via their terminals. This helps improve future meetings.

[0141] (Example 2)

[0142] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0143] Traditional meeting systems posed a significant challenge in creating meeting minutes by recording participants' speeches. This involved tedious tasks such as transcribing audio data into text and extracting key points, and it was difficult to adjust the meeting's pace to accommodate participants' emotional states and levels of understanding. As a result, meetings could become one-sided, potentially leading to misunderstandings or decreased motivation among some participants. This situation highlights the need for technology that can significantly improve meeting efficiency and participant engagement.

[0144] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0145] In this invention, the server includes a speech recognition means that recognizes speech in real time and converts it into text data, a key point extraction means that extracts key points from the text data using natural language processing and automatically generates meeting minutes, and an emotion recognition means that analyzes the facial expressions and voice tone of participants during the meeting to recognize their emotions. This makes it possible to improve the efficiency of meetings and provide an environment in which all participants can actively engage.

[0146] "Speech recognition means" refers to a function that processes speech data into text data in real time.

[0147] A "key point extraction method" is a function that extracts important information from text data and automatically compiles it into meeting minutes or reports.

[0148] "Emotion recognition means" refers to a function that analyzes participants' facial expressions and tone of voice to identify their emotional state.

[0149] A "meeting minutes distribution method" is a function that sends the generated meeting minutes to another device, making them accessible to all participants.

[0150] A "task management system" is a function for listing tasks that arise during a meeting and managing their progress.

[0151] A "meeting management tool" is a function that dynamically adjusts the progress of a meeting according to the participants' emotions and level of understanding, thereby promoting smooth communication.

[0152] A "schedule adjustment tool" is a function that optimizes meeting schedules based on date information and provides necessary information immediately.

[0153] "Means of providing information" refers to a function that provides timely reference materials related to the ongoing meeting, thereby supporting participants' understanding.

[0154] A "progress management method" is a function that allows managers to individually check the progress of selected tasks by specifying deadlines and responsible persons.

[0155] This system provides a program that integrates speech recognition, natural language processing, and emotion recognition technologies to improve meeting efficiency and promote active participant engagement. The system primarily consists of server, terminal, and user elements.

[0156] First, when a user starts a meeting, the device captures audio and video data in real time using its built-in microphone and camera. The device immediately sends this data to the server. To maintain secure communication, the device is required to use security protocols such as HTTPS for data transfer.

[0157] Next, the server uses speech recognition to convert the audio data into text data. This process employs common speech recognition technologies; for example, a cloud-based speech recognition API can be used. The server also uses emotion recognition on the video data to identify participants' emotions from their facial expressions and tone of voice.

[0158] The server applies natural language processing techniques to the converted text data, extracting key points and automatically generating meeting minutes. This process utilizes widely available natural language processing APIs. In parallel, it can also adjust the meeting's progress based on sentiment recognition; for example, if it analyzes that participants are confused, it can provide additional explanations or supplementary materials.

[0159] Users can view meeting minutes generated in real time and tasks performed during the meeting through their devices. This allows all participants to understand the meeting content and obtain necessary information in a timely manner.

[0160] As a concrete example, consider a scenario where a user initiates a corporate strategy meeting, and their device transmits audio and video to a server. Based on the information received, the server evaluates the level of understanding of all participants and, if interest wanes, suggests readjusting the flow of the meeting. In this way, the quality of communication is improved, making the meeting more effective.

[0161] When using a generative AI model, you can input text-based instructions as an example of a prompt, such as "Based on the participant sentiment analysis results, please formulate the agenda for the next meeting."

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] When a user starts a meeting, the device activates its microphone and camera and captures audio and video data in real time. The input consists of the voices and facial expressions of the meeting participants. The device uses digital signal processing to transform the analog data and outputs it as digital audio and video data.

[0165] Step 2:

[0166] The terminal sends the captured digital audio and video data to the server. The input is the digital data generated in step 1. For output, secure data transfer via protocols such as HTTPS takes place, and the data reaches the server.

[0167] Step 3:

[0168] The server applies a speech recognition module to the received audio data, converting the audio into text data. The input is digital audio data sent from the terminal. Speech recognition technology is used to process the data and generate text data as output.

[0169] Step 4:

[0170] The server analyzes the video data and identifies emotions using the participants' facial expressions and voice tone. The input is the digital video data transmitted in step 2. Facial feature points are extracted, and a machine learning algorithm is used to obtain an estimated result of the emotional state as output.

[0171] Step 5:

[0172] The server applies natural language processing to the converted text data, summarizing key information and automatically generating meeting minutes. The input is the text data obtained in step 3. It performs data processing using keyword extraction and summarization algorithms to generate meeting minutes as output.

[0173] Step 6:

[0174] The server dynamically adjusts the meeting's progress using the results of emotion recognition. The input is the emotional state data obtained in step 4. If a participant is determined to be confused, adjustments are made to the meeting, such as presenting additional materials. The output is fed back to the terminal as an optimized meeting plan and alert information.

[0175] Step 7:

[0176] Users can view meeting minutes and task lists generated through their terminals and track their work in real time as needed. Inputs consist of meeting minutes data and task data generated during the meeting, sent from the server to the terminal. Outputs are documented information displayed in the user interface, making it easier for users to understand the meeting content.

[0177] (Application Example 2)

[0178] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0179] Conventional meeting systems often fail to consider participants' emotional states when providing information or conducting meetings, leading to decreased understanding and engagement. Furthermore, insufficient automated support for meeting efficiency often resulted in one-sided discussions, making it difficult to create an environment where all participants could actively engage. This invention aims to solve these problems and realize communication support that reflects participants' emotions.

[0180] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0181] In this invention, the server includes: speech recognition means for recognizing speech in real time and converting it into text data; key point extraction means for extracting key points from the text data using natural language processing and automatically generating meeting minutes; meeting minute distribution means for distributing the generated meeting minutes to a communication device; task management means for automatically listing tasks that arise during a meeting and managing those tasks; schedule adjustment means for adjusting the meeting schedule based on date information and providing necessary information immediately; emotions analysis means; dynamic adjustment of the conversation based on the analyzed emotions; and means for providing information in accordance with the emotions of the participants. This makes it possible to conduct a meeting while considering the emotional state of the participants and to provide an efficient meeting environment in which all participants can actively participate.

[0182] "Speech recognition means" refers to technology that recognizes speech in real time and converts it into text data.

[0183] A "key point extraction method" is a technology that uses natural language processing to extract important information from text data and automatically generate meeting minutes.

[0184] "Meeting minutes distribution method" refers to the technology for transmitting generated meeting minutes to a communication device.

[0185] A "task management method" is a technology that automatically lists and manages tasks that arise during a meeting.

[0186] A "scheduling adjustment method" is a technology that adjusts meeting schedules based on information about dates and provides necessary information immediately.

[0187] "Methods for analyzing emotions" refer to technologies that analyze participants' emotions from data such as audio and video.

[0188] "Methods for dynamically adjusting the progress of a conversation" refer to techniques that adjust the way a conversation progresses in a timely manner based on analyzed emotions.

[0189] "Methods for providing information tailored to participants' emotions" refers to techniques for providing appropriate information according to the emotional state of the participants.

[0190] This invention is implemented through a system that combines speech recognition technology, natural language processing technology, and sentiment analysis technology. Specifically, a terminal captures audio and video in real time as soon as a meeting begins. The terminal sends the captured data to a server. The server uses a speech recognition library to convert the audio into text data. In this case, "speech_recognition" is used as a specific example. Important information is extracted from this text data using natural language processing technology. "NLPProcessor" is used for this technology.

[0191] The server analyzes the generated text data and uses an emotion engine to analyze participants' emotions. This process employs "EmotionRecognizer." Based on the results, meeting minutes are automatically generated and distributed in real time. Furthermore, the conversation is dynamically adjusted based on participants' emotions, and information is provided as needed. Task management technology is also applied to list and manage tasks that arise during the meeting.

[0192] A concrete example of its use is in a workplace project meeting. If the emotion engine detects a decline in participants' interest, the server can immediately provide additional materials or explanations to rekindle their interest. This ensures an environment where everyone remains actively engaged in the meeting.

[0193] By using a generative AI model to provide information tailored to the participant's emotions, the following prompt can be used: "If a participant is perceived to be showing anxiety during a discussion about travel plans, what information should be provided?"

[0194] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0195] Step 1:

[0196] The terminal captures audio and video data in real time using its microphone and camera as soon as the meeting begins. It acquires audio and video data as input and converts them into a digital format. This data is then sent to the server for further processing.

[0197] Step 2:

[0198] The server converts received audio data into text data using the "speech_recognition" library. It receives an audio file as input and formats it into a string in real time using speech recognition technology. The output, as text data, is used for subsequent natural language processing.

[0199] Step 3:

[0200] The server analyzes the converted text data using an "NLP Processor" to extract important information. It receives text data as input and applies natural language processing algorithms to identify key points. The output is key information for use in generating meeting minutes.

[0201] Step 4:

[0202] The server automatically generates meeting minutes from the key information obtained above and distributes them to the communication device. It receives key information as input, applies an automatic format, and creates formatted meeting minutes. The output is meeting minutes data accessible to participants.

[0203] Step 5:

[0204] The server simultaneously analyzes participants' emotions using the "EmotionRecognizer" emotion engine based on video data. The input is video data, and emotions are analyzed through facial recognition and voice tone analysis. The output is information about the participants' emotional state.

[0205] Step 6:

[0206] The server dynamically adjusts the flow of the conversation based on emotional information and selects information and additional explanations to provide to participants. It takes emotional states and meeting minutes as input and determines the content and timing of information provision accordingly. The output is the selected information and any additional materials presented.

[0207] Step 7:

[0208] Users review meeting minutes and additional information delivered from the server via their terminals and take necessary actions. The input consists of delivered data, allowing users to make decisions and execute subsequent steps and tasks based on their own judgment.

[0209] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0210] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0211] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0212] [Second Embodiment]

[0213] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0214] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0215] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0216] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0217] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0218] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0219] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0220] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0221] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0222] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0223] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0224] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0225] This invention is a system that improves the efficiency of meetings by integrating speech recognition technology and natural language processing technology. Users start a meeting using a meeting terminal. The terminal captures the audio during the meeting in real time via a microphone and transmits the audio data to a server.

[0226] The server quickly converts audio data into text data using a speech recognition module. This converted text data is then processed by a natural language processing module on the server to extract key points and generate an outline of the meeting minutes. The meeting minutes are automatically generated by the server and immediately delivered to the terminal. Users can review the minutes on their terminal and add comments as needed.

[0227] Furthermore, the server automatically lists any new tasks that arise during the meeting. Each task is accompanied by information such as a deadline and the person responsible, and is managed within the server. Users can check this task information from their terminals and manage its progress.

[0228] Furthermore, the server adjusts meeting schedules based on the user's schedule information. It also instantly searches for relevant documents and data as needed and provides them to the user's terminal. This ensures that users can access information without delay during meetings and focus on the discussion.

[0229] As a concrete example, consider a user holding a monthly reporting meeting. The user's device connects to the server at the start of the meeting and transmits what is said during the meeting to the server in real time. The server transcribes the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion about cost reduction takes place, the content is summarized and immediately shared with all participants.

[0230] This invention significantly improves meeting productivity and provides a system that helps users concentrate on the discussion.

[0231] The following describes the processing flow.

[0232] Step 1:

[0233] The terminal captures audio via its microphone at the start of the meeting. The captured audio data is divided into packets in real time and sent to the server.

[0234] Step 2:

[0235] The server inputs the received audio packets into the speech recognition module, converting the audio into text data. This allows the audio information to be treated as text information.

[0236] Step 3:

[0237] The server sends the text data to a natural language processing module to extract important information and key points from the meeting. The extracted information is temporarily stored on the server.

[0238] Step 4:

[0239] The server automatically creates meeting minutes based on the extracted key points. The created meeting minutes data is then sent back to the terminal.

[0240] Step 5:

[0241] Users can review meeting minutes received on their devices and add comments or corrections as needed. The minutes can also be saved and shared with other participants.

[0242] Step 6:

[0243] The server automatically creates a list of tasks discussed during the meeting and adds deadlines and assignees to each task. The generated task list is then delivered to the terminals.

[0244] Step 7:

[0245] Users can view their task list on their device and manage their progress. The completion status of each task is synchronized with the server, making it easy to track overall progress.

[0246] Step 8:

[0247] The server integrates with the user's calendar data to schedule the next meeting. It also displays necessary documents and information on the user's device during the meeting, improving meeting efficiency.

[0248] (Example 1)

[0249] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0250] Modern meetings are becoming increasingly complex, and with the growing volume and processing speed of information, efficient information management and record-keeping are required to allow participants to focus on the discussion. However, manual creation of meeting minutes, task management, and provision of related materials are time-consuming and labor-intensive, potentially leading to decreased productivity. Therefore, there is a need for a means to properly manage meeting information in real time and to efficiently advance discussions.

[0251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0252] In this invention, the server includes speech recognition means, key point extraction means, and meeting minute distribution means. This makes it possible to convert audio during a meeting into text data in real time, automatically extract key points of important discussions to generate meeting minutes, and immediately distribute them to participants. This promotes the immediate sharing of information and helps participants concentrate on the discussion.

[0253] "Speech recognition means" refers to a technology or device for converting speech data into text data in real time.

[0254] A "key point extraction method" is a technique or device that uses natural language processing to extract important information or key points of discussion from text data.

[0255] "Meeting minutes distribution means" refers to a technology or device that distributes generated meeting minutes to information terminals via communication.

[0256] "Work item management means" refers to a technology or device for automatically listing work items that arise during a meeting and for managing those work items.

[0257] "Schedule adjustment means" refers to technology or equipment for adjusting meeting schedules based on time information and providing necessary information immediately.

[0258] "Means of providing information" refers to a technology or device that provides information related to an ongoing meeting in real time based on text data acquired by speech recognition means.

[0259] "Progress management means" refers to a technology or device for identifying deadlines and responsible persons for work items extracted by work item management means, and for individually managing their progress.

[0260] This invention is a system that combines speech recognition technology and natural language processing technology, designed to streamline meetings. Users initiate a meeting using a conference terminal. The terminal uses a high-sensitivity microphone to capture audio in real time during the meeting. The audio data is stored digitally and transmitted to a server via a secure protocol (e.g., TLS / SSL).

[0261] The server uses a speech recognition module (for example, Google Speech-to-Text API, a common speech recognition API) to quickly convert audio data into text data. Then, a natural language processing module (e.g., SpaCy or NLTK) is used to analyze the converted text data and extract the key points of the meeting. The automatically generated meeting minutes are then immediately delivered to terminals by the server.

[0262] Users can view meeting minutes on their devices and add comments as needed. In this process, the server has a function to list new tasks that arise during the meeting. These tasks are accompanied by deadlines and assigned personnel information generated by an AI model, and are viewed and managed through the device. Furthermore, the server adjusts meeting schedules based on the user's time information and instantly searches for and provides necessary materials and data. Therefore, users can access necessary information without delay, even during meetings, enabling efficient discussions.

[0263] As a concrete example, consider a scenario where a user holds a regular progress report meeting. In this case, the user's device connects to the server at the start of the meeting and transmits audio to the server in real time. The server converts the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion takes place regarding the efficiency of management resources, the content is summarized and immediately shared with all participants.

[0264] An example of a prompt message would be, "Please extract the key points from the following audio and create meeting minutes." This would significantly improve meeting productivity and provide users with an environment where they can focus on the discussion.

[0265] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0266] Step 1:

[0267] Users initiate a meeting using a conference terminal. The terminal captures the meeting audio in real time using its built-in high-sensitivity microphone. The input is the audio from the meeting, and the output is a digital audio file (e.g., WAV format). The terminal temporarily saves the audio data to a PC or online storage to prevent data loss.

[0268] Step 2:

[0269] The terminal sends the stored audio data to the server using a secure protocol (e.g., TLS / SSL). This process involves data encryption. The input is the captured audio data, and the output is the encrypted audio data. This ensures secure data transfer.

[0270] Step 3:

[0271] The server converts received audio data into text data using a speech recognition module (e.g., a common speech recognition API). The input is the received audio data, and the output is text data (e.g., in text format). The server analyzes the audio waveform data, recognizes phonemes and words, and performs this conversion.

[0272] Step 4:

[0273] The server analyzes the generated text data using a natural language processing module (e.g., SpaCy) to extract the key points of the meeting. The input is the converted text data, and the output is a summary of the key points (e.g., summary text). The server calculates keyword frequency and sentence weighting within the text to extract important information.

[0274] Step 5:

[0275] The server automatically generates meeting minutes based on extracted key points. The input is data summarizing the key points, and the output is a formatted meeting minute (e.g., PDF or DOC format). This generation process utilizes a template engine and style processing. The server then distributes the generated meeting minutes to information terminals.

[0276] Step 6:

[0277] The server automatically lists work items that arise during a meeting using a generation AI model. The input is text data of the meeting content, and the output is a list of work items. The server analyzes the command structure and task-related expressions within the text to identify the items.

[0278] Step 7:

[0279] Users can review meeting minutes and work items delivered via their terminals and add comments and corrections. Inputs are the delivered meeting minutes and work items, while outputs are supplementary meeting minutes and an updated work item list. Users can manage their progress through their terminals.

[0280] (Application Example 1)

[0281] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0282] In a situation where emergency response is required, there is a problem that it is difficult to immediately and accurately aggregate the on-site voice information and quickly and accurately transmit it to the command center or the response team. In particular, it is required to efficiently judge the importance of the information generated on-site, quickly extract the necessary information, and share it.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0284] In this invention, the server includes an acoustic recognition means, an information extraction means, and an information aggregation means. Thereby, in the emergency response site, it is possible to quickly convert voice information into text, automatically extract important information, and immediately provide a report to the command center.

[0285] The "acoustic recognition means" is a means for acquiring voice as a digital signal and converting it into language data in real time.

[0286] The "information extraction means" is a means for extracting important features and keywords from the acquired language data using natural language processing and automatically generating a record.

[0287] The "recording distribution means" is a means for distributing the generated record to a communication device and sharing it with relevant parties.

[0288] The "work management means" is a means for automatically listing up the work generated during a meeting or response and managing it.

[0289] The "schedule adjustment means" is a means for adjusting the schedule of a meeting based on schedule information and immediately providing necessary information.

[0290] An "information aggregation method" is a means of aggregating on-site audio information, extracting important keywords, and generating an automated report in situations requiring action.

[0291] One embodiment of this invention consists of a system integrating acoustic recognition means, information extraction means, recording and distribution means, and the like. This system uses specific hardware and software to acquire sound in real time and convert it into language data.

[0292] Specifically, a device such as a smartphone captures audio data with its microphone and converts it into language data using the Google Cloud Speech-to-Text API. This speech recognition requires an acoustic model using TensorFlow.

[0293] The converted language data is sent to a server, where the SpaCy natural language processing library is used by an information extraction system to identify key keywords and automatically generate records. The generated records are immediately distributed to relevant parties using cloud communication capabilities. The information aggregation system quickly processes information that is particularly needed in emergencies and provides important content to command centers and other relevant locations.

[0294] In the operation of this system, the server lists tasks as they arise and provides a task management function that makes it easy for users to check their progress from their terminals. In addition, the scheduling mechanism allows for quick and efficient management of meeting dates, minimizing inconvenience caused by schedule changes.

[0295] For example, if keywords requiring emergency response, such as "fire," "smoke," and "evacuation," are identified at a fire response site, a report containing these keywords will be immediately generated and distributed. This enables rapid information sharing between the field and the command center.

[0296] An example of a prompt for the generating AI model would be the text: "Please tell me how to extract important keywords from audio data at an emergency response site and automatically generate a report to the command center immediately."

[0297] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0298] Step 1:

[0299] The device uses a microphone to capture audio data from the field and generates a digitized audio signal. This signal is sent to the Google Cloud Speech-to-Text API. The input is analog audio, and the output is a digitized audio signal.

[0300] Step 2:

[0301] The server uses the Google Cloud Speech-to-Text API to analyze the speech signal and convert it into language data in real time. An acoustic model is used for this conversion, sequentially outputting text data from the speech data. The input is a digitized speech signal, and the output is text data.

[0302] Step 3:

[0303] The server uses SpaCy to extract important keywords from text data. Natural language processing is employed here to clarify key features from the linguistic data. The input is text data, and the output consists of extracted keywords and important content.

[0304] Step 4:

[0305] As a means of information aggregation, the server instantly generates important reports based on extracted keywords. These reports contain all the necessary content and are quickly transmitted to the command center. The input is the important information, and the output is the generated report.

[0306] Step 5:

[0307] The user views the report generated on the terminal and reports the situation to the command center as needed. By the user checking the report, smooth communication becomes possible. The input is the generated report, and the output is the user's actions and feedback.

[0308] Through this process, the generative AI model is utilized to achieve the immediate processing and transmission of information at the emergency site.

[0309] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion recognition model 59 and perform specific processing using the user's emotions.

[0310] This invention is a system that incorporates an emotion engine for recognizing the user's emotions in addition to voice recognition technology and natural language processing technology, improving the efficiency of meetings and the engagement of participants. The system is centered around three elements: the server, the terminal, and the user.

[0311] When the user starts a meeting, the terminal uses the microphone and camera to capture audio and video data in real time. The terminal transmits the captured data to the server. The server uses the voice recognition module to convert the voice into character data, and at the same time analyzes the user's expressions and voice tones from the video data to recognize emotions.

[0312] The server uses the natural language processing module to extract important information from the character data, summarizes the key points of the meeting, and generates meeting minutes. At the same time, the emotion engine determines the user's emotional state, and based on that information, the progress of the meeting can be dynamically adjusted. For example, when it is determined that the participants are confused, an alert for providing additional explanations and relevant materials is displayed on the terminal.

[0313] Users can view meeting minutes generated on their devices, and tasks that arise during the meeting are displayed in real time. The server assists in optimizing the meeting's outcome based on user sentiment information. This creates an environment where discussions are not one-sided and all participants can actively engage.

[0314] As a concrete example, when holding a strategy meeting, the user activates a terminal to start the meeting. The terminal captures audio and video, and the server performs analysis. If a presentation is given during the meeting and the emotion engine detects a decline in participants' interest or understanding, the server suggests options to readjust the focus of the meeting based on that information. In this way, this invention functions not only as a means of transmitting information, but also as a tool to improve the quality of communication.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The device captures audio and video using its built-in microphone and camera as soon as the meeting starts. The captured data is sent to the server in real time.

[0318] Step 2:

[0319] The server converts the audio data received by the speech recognition module into text data. This process transcribes the audio content into text.

[0320] Step 3:

[0321] The server simultaneously analyzes the video data using an emotion engine, recognizing the user's emotional state from their facial expressions, facial movements, and voice tone. Changes in emotion and specific states are identified.

[0322] Step 4:

[0323] The server analyzes text data through a natural language processing module and extracts important information and key points from the meeting. Based on the extraction results, meeting minutes are automatically generated.

[0324] Step 5:

[0325] Based on the emotional information detected by the emotion engine, the server analyzes the user's level of understanding and reactions, and provides suggestions for meeting progress or additional materials if necessary. This information is then notified to the terminal.

[0326] Step 6:

[0327] Users can review the meeting minutes and suggested supplementary information generated on their devices, allowing them to gain a deeper understanding of the meeting content and participate in the discussion.

[0328] Step 7:

[0329] The server automatically lists any new tasks that arise during the meeting, identifying deadlines and assigning responsibilities. The generated task list is also delivered to the user's terminal, allowing them to monitor progress.

[0330] Step 8:

[0331] Based on user feedback, the server prepares post-meeting feedback and provides it to users via their terminals. This helps improve future meetings.

[0332] (Example 2)

[0333] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0334] Traditional meeting systems posed a significant challenge in creating meeting minutes by recording participants' speeches. This involved tedious tasks such as transcribing audio data into text and extracting key points, and it was difficult to adjust the meeting's pace to accommodate participants' emotional states and levels of understanding. As a result, meetings could become one-sided, potentially leading to misunderstandings or decreased motivation among some participants. This situation highlights the need for technology that can significantly improve meeting efficiency and participant engagement.

[0335] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0336] In this invention, the server includes a speech recognition means that recognizes speech in real time and converts it into text data, a key point extraction means that extracts key points from the text data using natural language processing and automatically generates meeting minutes, and an emotion recognition means that analyzes the facial expressions and voice tone of participants during the meeting to recognize their emotions. This makes it possible to improve the efficiency of meetings and provide an environment in which all participants can actively engage.

[0337] "Speech recognition means" refers to a function that processes speech data into text data in real time.

[0338] A "key point extraction method" is a function that extracts important information from text data and automatically compiles it into meeting minutes or reports.

[0339] "Emotion recognition means" refers to a function that analyzes participants' facial expressions and tone of voice to identify their emotional state.

[0340] A "meeting minutes distribution method" is a function that sends the generated meeting minutes to another device, making them accessible to all participants.

[0341] A "task management system" is a function for listing tasks that arise during a meeting and managing their progress.

[0342] A "meeting management tool" is a function that dynamically adjusts the progress of a meeting according to the participants' emotions and level of understanding, thereby promoting smooth communication.

[0343] A "schedule adjustment tool" is a function that optimizes meeting schedules based on date information and provides necessary information immediately.

[0344] "Means of providing information" refers to a function that provides timely reference materials related to the ongoing meeting, thereby supporting participants' understanding.

[0345] A "progress management method" is a function that allows managers to individually check the progress of selected tasks by specifying deadlines and responsible persons.

[0346] This system provides a program that integrates speech recognition, natural language processing, and emotion recognition technologies to improve meeting efficiency and promote active participant engagement. The system primarily consists of server, terminal, and user elements.

[0347] First, when a user starts a meeting, the device captures audio and video data in real time using its built-in microphone and camera. The device immediately sends this data to the server. To maintain secure communication, the device is required to use security protocols such as HTTPS for data transfer.

[0348] Next, the server uses speech recognition to convert the audio data into text data. This process employs common speech recognition technologies; for example, a cloud-based speech recognition API can be used. The server also uses emotion recognition on the video data to identify participants' emotions from their facial expressions and tone of voice.

[0349] The server applies natural language processing techniques to the converted text data, extracting key points and automatically generating meeting minutes. This process utilizes widely available natural language processing APIs. In parallel, it can also adjust the meeting's progress based on sentiment recognition; for example, if it analyzes that participants are confused, it can provide additional explanations or supplementary materials.

[0350] Users can view meeting minutes generated in real time and tasks performed during the meeting through their devices. This allows all participants to understand the meeting content and obtain necessary information in a timely manner.

[0351] As a concrete example, consider a scenario where a user initiates a corporate strategy meeting, and their device transmits audio and video to a server. Based on the information received, the server evaluates the level of understanding of all participants and, if interest wanes, suggests readjusting the flow of the meeting. In this way, the quality of communication is improved, making the meeting more effective.

[0352] When using a generative AI model, you can input text-based instructions as an example of a prompt, such as "Based on the participant sentiment analysis results, please formulate the agenda for the next meeting."

[0353] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0354] Step 1:

[0355] When a user starts a meeting, the device activates its microphone and camera and captures audio and video data in real time. The input consists of the voices and facial expressions of the meeting participants. The device uses digital signal processing to transform the analog data and outputs it as digital audio and video data.

[0356] Step 2:

[0357] The terminal sends the captured digital audio and video data to the server. The input is the digital data generated in step 1. For output, secure data transfer via protocols such as HTTPS takes place, and the data reaches the server.

[0358] Step 3:

[0359] The server applies a speech recognition module to the received audio data, converting the audio into text data. The input is digital audio data sent from the terminal. Speech recognition technology is used to process the data and generate text data as output.

[0360] Step 4:

[0361] The server analyzes the video data and identifies emotions using the participants' facial expressions and voice tone. The input is the digital video data transmitted in step 2. Facial feature points are extracted, and a machine learning algorithm is used to obtain an estimated result of the emotional state as output.

[0362] Step 5:

[0363] The server applies natural language processing to the converted text data, summarizing key information and automatically generating meeting minutes. The input is the text data obtained in step 3. It performs data processing using keyword extraction and summarization algorithms to generate meeting minutes as output.

[0364] Step 6:

[0365] The server dynamically adjusts the meeting's progress using the results of emotion recognition. The input is the emotional state data obtained in step 4. If a participant is determined to be confused, adjustments are made to the meeting, such as presenting additional materials. The output is fed back to the terminal as an optimized meeting plan and alert information.

[0366] Step 7:

[0367] Users can view meeting minutes and task lists generated through their terminals and track their work in real time as needed. Inputs consist of meeting minutes data and task data generated during the meeting, sent from the server to the terminal. Outputs are documented information displayed in the user interface, making it easier for users to understand the meeting content.

[0368] (Application Example 2)

[0369] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0370] Conventional meeting systems often fail to consider participants' emotional states when providing information or conducting meetings, leading to decreased understanding and engagement. Furthermore, insufficient automated support for meeting efficiency often resulted in one-sided discussions, making it difficult to create an environment where all participants could actively engage. This invention aims to solve these problems and realize communication support that reflects participants' emotions.

[0371] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0372] In this invention, the server includes: speech recognition means for recognizing speech in real time and converting it into text data; key point extraction means for extracting key points from the text data using natural language processing and automatically generating meeting minutes; meeting minute distribution means for distributing the generated meeting minutes to a communication device; task management means for automatically listing tasks that arise during a meeting and managing those tasks; schedule adjustment means for adjusting the meeting schedule based on date information and providing necessary information immediately; emotions analysis means; dynamic adjustment of the conversation based on the analyzed emotions; and means for providing information in accordance with the emotions of the participants. This makes it possible to conduct a meeting while considering the emotional state of the participants and to provide an efficient meeting environment in which all participants can actively participate.

[0373] "Speech recognition means" refers to technology that recognizes speech in real time and converts it into text data.

[0374] A "key point extraction method" is a technology that uses natural language processing to extract important information from text data and automatically generate meeting minutes.

[0375] "Meeting minutes distribution method" refers to the technology for transmitting generated meeting minutes to a communication device.

[0376] A "task management method" is a technology that automatically lists and manages tasks that arise during a meeting.

[0377] A "scheduling adjustment method" is a technology that adjusts meeting schedules based on information about dates and provides necessary information immediately.

[0378] "Methods for analyzing emotions" refer to technologies that analyze participants' emotions from data such as audio and video.

[0379] "Methods for dynamically adjusting the progress of a conversation" refer to techniques that adjust the way a conversation progresses in a timely manner based on analyzed emotions.

[0380] "Methods for providing information tailored to participants' emotions" refers to techniques for providing appropriate information according to the emotional state of the participants.

[0381] This invention is implemented through a system that combines speech recognition technology, natural language processing technology, and sentiment analysis technology. Specifically, a terminal captures audio and video in real time as soon as a meeting begins. The terminal sends the captured data to a server. The server uses a speech recognition library to convert the audio into text data. In this case, "speech_recognition" is used as a specific example. Important information is extracted from this text data using natural language processing technology. "NLPProcessor" is used for this technology.

[0382] The server analyzes the generated text data and uses an emotion engine to analyze participants' emotions. This process employs "EmotionRecognizer." Based on the results, meeting minutes are automatically generated and distributed in real time. Furthermore, the conversation is dynamically adjusted based on participants' emotions, and information is provided as needed. Task management technology is also applied to list and manage tasks that arise during the meeting.

[0383] A concrete example of its use is in a workplace project meeting. If the emotion engine detects a decline in participants' interest, the server can immediately provide additional materials or explanations to rekindle their interest. This ensures an environment where everyone remains actively engaged in the meeting.

[0384] By using a generative AI model to provide information tailored to the participant's emotions, the following prompt can be used: "If a participant is perceived to be showing anxiety during a discussion about travel plans, what information should be provided?"

[0385] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0386] Step 1:

[0387] The terminal captures audio and video data in real time using its microphone and camera as soon as the meeting begins. It acquires audio and video data as input and converts them into a digital format. This data is then sent to the server for further processing.

[0388] Step 2:

[0389] The server converts received audio data into text data using the "speech_recognition" library. It receives an audio file as input and formats it into a string in real time using speech recognition technology. The output, as text data, is used for subsequent natural language processing.

[0390] Step 3:

[0391] The server analyzes the converted text data using an "NLP Processor" to extract important information. It receives text data as input and applies natural language processing algorithms to identify key points. The output is key information for use in generating meeting minutes.

[0392] Step 4:

[0393] The server automatically generates meeting minutes from the key information obtained above and distributes them to the communication device. It receives key information as input, applies an automatic format, and creates formatted meeting minutes. The output is meeting minutes data accessible to participants.

[0394] Step 5:

[0395] The server simultaneously analyzes participants' emotions using the "EmotionRecognizer" emotion engine based on video data. The input is video data, and emotions are analyzed through facial recognition and voice tone analysis. The output is information about the participants' emotional state.

[0396] Step 6:

[0397] The server dynamically adjusts the flow of the conversation based on emotional information and selects information and additional explanations to provide to participants. It takes emotional states and meeting minutes as input and determines the content and timing of information provision accordingly. The output is the selected information and any additional materials presented.

[0398] Step 7:

[0399] Users review meeting minutes and additional information delivered from the server via their terminals and take necessary actions. The input consists of delivered data, allowing users to make decisions and execute subsequent steps and tasks based on their own judgment.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0401] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0403] [Third Embodiment]

[0404] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0405] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0406] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0408] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0410] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0411] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0412] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0415] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0416] This invention is a system that improves the efficiency of meetings by integrating speech recognition technology and natural language processing technology. Users start a meeting using a meeting terminal. The terminal captures the audio during the meeting in real time via a microphone and transmits the audio data to a server.

[0417] The server quickly converts audio data into text data using a speech recognition module. This converted text data is then processed by a natural language processing module on the server to extract key points and generate an outline of the meeting minutes. The meeting minutes are automatically generated by the server and immediately delivered to the terminal. Users can review the minutes on their terminal and add comments as needed.

[0418] Furthermore, the server automatically lists any new tasks that arise during the meeting. Each task is accompanied by information such as a deadline and the person responsible, and is managed within the server. Users can check this task information from their terminals and manage its progress.

[0419] Furthermore, the server adjusts meeting schedules based on the user's schedule information. It also instantly searches for relevant documents and data as needed and provides them to the user's terminal. This ensures that users can access information without delay during meetings and focus on the discussion.

[0420] As a concrete example, consider a user holding a monthly reporting meeting. The user's device connects to the server at the start of the meeting and transmits what is said during the meeting to the server in real time. The server transcribes the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion about cost reduction takes place, the content is summarized and immediately shared with all participants.

[0421] This invention significantly improves meeting productivity and provides a system that helps users concentrate on the discussion.

[0422] The following describes the processing flow.

[0423] Step 1:

[0424] The terminal captures audio via its microphone at the start of the meeting. The captured audio data is divided into packets in real time and sent to the server.

[0425] Step 2:

[0426] The server inputs the received audio packets into the speech recognition module, converting the audio into text data. This allows the audio information to be treated as text information.

[0427] Step 3:

[0428] The server sends the text data to a natural language processing module to extract important information and key points from the meeting. The extracted information is temporarily stored on the server.

[0429] Step 4:

[0430] The server automatically creates meeting minutes based on the extracted key points. The created meeting minutes data is then sent back to the terminal.

[0431] Step 5:

[0432] Users can review meeting minutes received on their devices and add comments or corrections as needed. The minutes can also be saved and shared with other participants.

[0433] Step 6:

[0434] The server automatically creates a list of tasks discussed during the meeting and adds deadlines and assignees to each task. The generated task list is then delivered to the terminals.

[0435] Step 7:

[0436] Users can view their task list on their device and manage their progress. The completion status of each task is synchronized with the server, making it easy to track overall progress.

[0437] Step 8:

[0438] The server integrates with the user's calendar data to schedule the next meeting. It also displays necessary documents and information on the user's device during the meeting, improving meeting efficiency.

[0439] (Example 1)

[0440] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0441] Modern meetings are becoming increasingly complex, and with the growing volume and processing speed of information, efficient information management and record-keeping are required to allow participants to focus on the discussion. However, manual creation of meeting minutes, task management, and provision of related materials are time-consuming and labor-intensive, potentially leading to decreased productivity. Therefore, there is a need for a means to properly manage meeting information in real time and to efficiently advance discussions.

[0442] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0443] In this invention, the server includes speech recognition means, key point extraction means, and meeting minute distribution means. This makes it possible to convert audio during a meeting into text data in real time, automatically extract key points of important discussions to generate meeting minutes, and immediately distribute them to participants. This promotes the immediate sharing of information and helps participants concentrate on the discussion.

[0444] "Speech recognition means" refers to a technology or device for converting speech data into text data in real time.

[0445] A "key point extraction method" is a technique or device that uses natural language processing to extract important information or key points of discussion from text data.

[0446] "Meeting minutes distribution means" refers to a technology or device that distributes generated meeting minutes to information terminals via communication.

[0447] "Work item management means" refers to a technology or device for automatically listing work items that arise during a meeting and for managing those work items.

[0448] "Schedule adjustment means" refers to technology or equipment for adjusting meeting schedules based on time information and providing necessary information immediately.

[0449] "Means of providing information" refers to a technology or device that provides information related to an ongoing meeting in real time based on text data acquired by speech recognition means.

[0450] "Progress management means" refers to a technology or device for identifying deadlines and responsible persons for work items extracted by work item management means, and for individually managing their progress.

[0451] This invention is a system that combines speech recognition technology and natural language processing technology, designed to streamline meetings. Users initiate a meeting using a conference terminal. The terminal uses a high-sensitivity microphone to capture audio in real time during the meeting. The audio data is stored digitally and transmitted to a server via a secure protocol (e.g., TLS / SSL).

[0452] The server uses a speech recognition module (for example, Google Speech-to-Text API, a common speech recognition API) to quickly convert audio data into text data. Then, a natural language processing module (e.g., SpaCy or NLTK) is used to analyze the converted text data and extract the key points of the meeting. The automatically generated meeting minutes are then immediately delivered to terminals by the server.

[0453] Users can view meeting minutes on their devices and add comments as needed. In this process, the server has a function to list new tasks that arise during the meeting. These tasks are accompanied by deadlines and assigned personnel information generated by an AI model, and are viewed and managed through the device. Furthermore, the server adjusts meeting schedules based on the user's time information and instantly searches for and provides necessary materials and data. Therefore, users can access necessary information without delay, even during meetings, enabling efficient discussions.

[0454] As a concrete example, consider a scenario where a user holds a regular progress report meeting. In this case, the user's device connects to the server at the start of the meeting and transmits audio to the server in real time. The server converts the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion takes place regarding the efficiency of management resources, the content is summarized and immediately shared with all participants.

[0455] An example of a prompt message would be, "Please extract the key points from the following audio and create meeting minutes." This would significantly improve meeting productivity and provide users with an environment where they can focus on the discussion.

[0456] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0457] Step 1:

[0458] Users initiate a meeting using a conference terminal. The terminal captures the meeting audio in real time using its built-in high-sensitivity microphone. The input is the audio from the meeting, and the output is a digital audio file (e.g., WAV format). The terminal temporarily saves the audio data to a PC or online storage to prevent data loss.

[0459] Step 2:

[0460] The terminal sends the stored audio data to the server using a secure protocol (e.g., TLS / SSL). This process involves data encryption. The input is the captured audio data, and the output is the encrypted audio data. This ensures secure data transfer.

[0461] Step 3:

[0462] The server converts received audio data into text data using a speech recognition module (e.g., a common speech recognition API). The input is the received audio data, and the output is text data (e.g., in text format). The server analyzes the audio waveform data, recognizes phonemes and words, and performs this conversion.

[0463] Step 4:

[0464] The server analyzes the generated text data using a natural language processing module (e.g., SpaCy) to extract the key points of the meeting. The input is the converted text data, and the output is a summary of the key points (e.g., summary text). The server calculates keyword frequency and sentence weighting within the text to extract important information.

[0465] Step 5:

[0466] The server automatically generates meeting minutes based on extracted key points. The input is data summarizing the key points, and the output is a formatted meeting minute (e.g., PDF or DOC format). This generation process utilizes a template engine and style processing. The server then distributes the generated meeting minutes to information terminals.

[0467] Step 6:

[0468] The server automatically lists work items that arise during a meeting using a generation AI model. The input is text data of the meeting content, and the output is a list of work items. The server analyzes the command structure and task-related expressions within the text to identify the items.

[0469] Step 7:

[0470] Users can review meeting minutes and work items delivered via their terminals and add comments and corrections. Inputs are the delivered meeting minutes and work items, while outputs are supplementary meeting minutes and an updated work item list. Users can manage their progress through their terminals.

[0471] (Application Example 1)

[0472] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] In emergency situations, there is a problem in quickly and accurately gathering on-site audio information and transmitting it to the command center and response team with speed and accuracy. In particular, it is necessary to efficiently assess the importance of information generated on-site, quickly extract the necessary information, and share it.

[0474] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0475] In this invention, the server includes an acoustic recognition means, an information extraction means, and an information aggregation means. This makes it possible to quickly convert voice information into text, automatically extract important information, and immediately provide a report to the command center at the scene of an emergency response.

[0476] "Acoustic recognition means" refers to a means of acquiring sound as a digital signal and converting it into language data in real time.

[0477] An "information extraction method" is a means of extracting important features and keywords from acquired language data using natural language processing and automatically generating records.

[0478] A "record distribution method" is a means of distributing generated records to communication devices and sharing them with relevant parties.

[0479] A "task management system" is a means of automatically listing and managing tasks that arise during meetings or interactions.

[0480] A "schedule adjustment method" is a means of adjusting meeting schedules based on date information and providing necessary information immediately.

[0481] An "information aggregation method" is a means of aggregating on-site audio information, extracting important keywords, and generating an automated report in situations requiring action.

[0482] One embodiment of this invention consists of a system integrating acoustic recognition means, information extraction means, recording and distribution means, and the like. This system uses specific hardware and software to acquire sound in real time and convert it into language data.

[0483] Specifically, a device such as a smartphone captures audio data with its microphone and converts it into language data using the Google Cloud Speech-to-Text API. This speech recognition requires an acoustic model using TensorFlow.

[0484] The converted language data is sent to a server, where the SpaCy natural language processing library is used by an information extraction system to identify key keywords and automatically generate records. The generated records are immediately distributed to relevant parties using cloud communication capabilities. The information aggregation system quickly processes information that is particularly needed in emergencies and provides important content to command centers and other relevant locations.

[0485] In the operation of this system, the server lists tasks as they arise and provides a task management function that makes it easy for users to check their progress from their terminals. In addition, the scheduling mechanism allows for quick and efficient management of meeting dates, minimizing inconvenience caused by schedule changes.

[0486] For example, if keywords requiring emergency response, such as "fire," "smoke," and "evacuation," are identified at a fire response site, a report containing these keywords will be immediately generated and distributed. This enables rapid information sharing between the field and the command center.

[0487] An example of a prompt for the generating AI model would be the text: "Please tell me how to extract important keywords from audio data at an emergency response site and automatically generate a report to the command center immediately."

[0488] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0489] Step 1:

[0490] The device uses a microphone to capture audio data from the field and generates a digitized audio signal. This signal is sent to the Google Cloud Speech-to-Text API. The input is analog audio, and the output is a digitized audio signal.

[0491] Step 2:

[0492] The server uses the Google Cloud Speech-to-Text API to analyze the speech signal and convert it into language data in real time. An acoustic model is used for this conversion, sequentially outputting text data from the speech data. The input is a digitized speech signal, and the output is text data.

[0493] Step 3:

[0494] The server uses SpaCy to extract important keywords from text data. Natural language processing is employed here to clarify key features from the linguistic data. The input is text data, and the output consists of extracted keywords and important content.

[0495] Step 4:

[0496] As a means of information aggregation, the server instantly generates important reports based on extracted keywords. These reports contain all the necessary content and are quickly transmitted to the command center. The input is the important information, and the output is the generated report.

[0497] Step 5:

[0498] Users view reports generated on their devices and report the situation to the command center as needed. User review of the reports facilitates smoother communication. The input is the generated reports, and the output is the user's actions and feedback.

[0499] Through this process, we will utilize generative AI models to enable immediate processing and transmission of information at emergency sites.

[0500] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0501] This invention is a system that improves meeting efficiency and participant engagement by incorporating an emotion engine that recognizes user emotions, in addition to speech recognition technology and natural language processing technology. The system is composed of three main elements: a server, terminals, and users.

[0502] When a user starts a meeting, their device uses its microphone and camera to capture audio and video data in real time. The device sends the captured data to a server. The server uses a speech recognition module to convert the audio into text data, and simultaneously analyzes the user's facial expressions and tone of voice from the video data to recognize their emotions.

[0503] The server uses a natural language processing module to extract important information from text data, summarize the key points of the meeting, and generate meeting minutes. Simultaneously, an emotion engine identifies the user's emotional state, and the meeting's progress can be dynamically adjusted based on this information. For example, if a participant is deemed confused, an alert will be displayed on the terminal to provide additional explanations or relevant materials.

[0504] Users can view meeting minutes generated on their devices, and tasks that arise during the meeting are displayed in real time. The server assists in optimizing the meeting's outcome based on user sentiment information. This creates an environment where discussions are not one-sided and all participants can actively engage.

[0505] As a concrete example, when holding a strategy meeting, the user activates a terminal to start the meeting. The terminal captures audio and video, and the server performs analysis. If a presentation is given during the meeting and the emotion engine detects a decline in participants' interest or understanding, the server suggests options to readjust the focus of the meeting based on that information. In this way, this invention functions not only as a means of transmitting information, but also as a tool to improve the quality of communication.

[0506] The following describes the processing flow.

[0507] Step 1:

[0508] The device captures audio and video using its built-in microphone and camera as soon as the meeting starts. The captured data is sent to the server in real time.

[0509] Step 2:

[0510] The server converts the audio data received by the speech recognition module into text data. This process transcribes the audio content into text.

[0511] Step 3:

[0512] The server simultaneously analyzes the video data using an emotion engine, recognizing the user's emotional state from their facial expressions, facial movements, and voice tone. Changes in emotion and specific states are identified.

[0513] Step 4:

[0514] The server analyzes text data through a natural language processing module and extracts important information and key points from the meeting. Based on the extraction results, meeting minutes are automatically generated.

[0515] Step 5:

[0516] Based on the emotional information detected by the emotion engine, the server analyzes the user's level of understanding and reactions, and provides suggestions for meeting progress or additional materials if necessary. This information is then notified to the terminal.

[0517] Step 6:

[0518] Users can review the meeting minutes and suggested supplementary information generated on their devices, allowing them to gain a deeper understanding of the meeting content and participate in the discussion.

[0519] Step 7:

[0520] The server automatically lists any new tasks that arise during the meeting, identifying deadlines and assigning responsibilities. The generated task list is also delivered to the user's terminal, allowing them to monitor progress.

[0521] Step 8:

[0522] Based on user feedback, the server prepares post-meeting feedback and provides it to users via their terminals. This helps improve future meetings.

[0523] (Example 2)

[0524] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] Traditional meeting systems posed a significant challenge in creating meeting minutes by recording participants' speeches. This involved tedious tasks such as transcribing audio data into text and extracting key points, and it was difficult to adjust the meeting's pace to accommodate participants' emotional states and levels of understanding. As a result, meetings could become one-sided, potentially leading to misunderstandings or decreased motivation among some participants. This situation highlights the need for technology that can significantly improve meeting efficiency and participant engagement.

[0526] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0527] In this invention, the server includes a speech recognition means that recognizes speech in real time and converts it into text data, a key point extraction means that extracts key points from the text data using natural language processing and automatically generates meeting minutes, and an emotion recognition means that analyzes the facial expressions and voice tone of participants during the meeting to recognize their emotions. This makes it possible to improve the efficiency of meetings and provide an environment in which all participants can actively engage.

[0528] "Speech recognition means" refers to a function that processes speech data into text data in real time.

[0529] A "key point extraction method" is a function that extracts important information from text data and automatically compiles it into meeting minutes or reports.

[0530] "Emotion recognition means" refers to a function that analyzes participants' facial expressions and tone of voice to identify their emotional state.

[0531] A "meeting minutes distribution method" is a function that sends the generated meeting minutes to another device, making them accessible to all participants.

[0532] A "task management system" is a function for listing tasks that arise during a meeting and managing their progress.

[0533] A "meeting management tool" is a function that dynamically adjusts the progress of a meeting according to the participants' emotions and level of understanding, thereby promoting smooth communication.

[0534] A "schedule adjustment tool" is a function that optimizes meeting schedules based on date information and provides necessary information immediately.

[0535] "Means of providing information" refers to a function that provides timely reference materials related to the ongoing meeting, thereby supporting participants' understanding.

[0536] A "progress management method" is a function that allows managers to individually check the progress of selected tasks by specifying deadlines and responsible persons.

[0537] This system provides a program that integrates speech recognition, natural language processing, and emotion recognition technologies to improve meeting efficiency and promote active participant engagement. The system primarily consists of server, terminal, and user elements.

[0538] First, when a user starts a meeting, the device captures audio and video data in real time using its built-in microphone and camera. The device immediately sends this data to the server. To maintain secure communication, the device is required to use security protocols such as HTTPS for data transfer.

[0539] Next, the server uses speech recognition to convert the audio data into text data. This process employs common speech recognition technologies; for example, a cloud-based speech recognition API can be used. The server also uses emotion recognition on the video data to identify participants' emotions from their facial expressions and tone of voice.

[0540] The server applies natural language processing techniques to the converted text data, extracting key points and automatically generating meeting minutes. This process utilizes widely available natural language processing APIs. In parallel, it can also adjust the meeting's progress based on sentiment recognition; for example, if it analyzes that participants are confused, it can provide additional explanations or supplementary materials.

[0541] Users can view meeting minutes generated in real time and tasks performed during the meeting through their devices. This allows all participants to understand the meeting content and obtain necessary information in a timely manner.

[0542] As a concrete example, consider a scenario where a user initiates a corporate strategy meeting, and their device transmits audio and video to a server. Based on the information received, the server evaluates the level of understanding of all participants and, if interest wanes, suggests readjusting the flow of the meeting. In this way, the quality of communication is improved, making the meeting more effective.

[0543] When using a generative AI model, you can input text-based instructions as an example of a prompt, such as "Based on the participant sentiment analysis results, please formulate the agenda for the next meeting."

[0544] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0545] Step 1:

[0546] When a user starts a meeting, the device activates its microphone and camera and captures audio and video data in real time. The input consists of the voices and facial expressions of the meeting participants. The device uses digital signal processing to transform the analog data and outputs it as digital audio and video data.

[0547] Step 2:

[0548] The terminal sends the captured digital audio and video data to the server. The input is the digital data generated in step 1. For output, secure data transfer via protocols such as HTTPS takes place, and the data reaches the server.

[0549] Step 3:

[0550] The server applies a speech recognition module to the received audio data, converting the audio into text data. The input is digital audio data sent from the terminal. Speech recognition technology is used to process the data and generate text data as output.

[0551] Step 4:

[0552] The server analyzes the video data and identifies emotions using the participants' facial expressions and voice tone. The input is the digital video data transmitted in step 2. Facial feature points are extracted, and a machine learning algorithm is used to obtain an estimated result of the emotional state as output.

[0553] Step 5:

[0554] The server applies natural language processing to the converted text data, summarizing key information and automatically generating meeting minutes. The input is the text data obtained in step 3. It performs data processing using keyword extraction and summarization algorithms to generate meeting minutes as output.

[0555] Step 6:

[0556] The server dynamically adjusts the meeting's progress using the results of emotion recognition. The input is the emotional state data obtained in step 4. If a participant is determined to be confused, adjustments are made to the meeting, such as presenting additional materials. The output is fed back to the terminal as an optimized meeting plan and alert information.

[0557] Step 7:

[0558] Users can view meeting minutes and task lists generated through their terminals and track their work in real time as needed. Inputs consist of meeting minutes data and task data generated during the meeting, sent from the server to the terminal. Outputs are documented information displayed in the user interface, making it easier for users to understand the meeting content.

[0559] (Application Example 2)

[0560] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0561] Conventional meeting systems often fail to consider participants' emotional states when providing information or conducting meetings, leading to decreased understanding and engagement. Furthermore, insufficient automated support for meeting efficiency often resulted in one-sided discussions, making it difficult to create an environment where all participants could actively engage. This invention aims to solve these problems and realize communication support that reflects participants' emotions.

[0562] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0563] In this invention, the server includes: speech recognition means for recognizing speech in real time and converting it into text data; key point extraction means for extracting key points from the text data using natural language processing and automatically generating meeting minutes; meeting minute distribution means for distributing the generated meeting minutes to a communication device; task management means for automatically listing tasks that arise during a meeting and managing those tasks; schedule adjustment means for adjusting the meeting schedule based on date information and providing necessary information immediately; emotions analysis means; dynamic adjustment of the conversation based on the analyzed emotions; and means for providing information in accordance with the emotions of the participants. This makes it possible to conduct a meeting while considering the emotional state of the participants and to provide an efficient meeting environment in which all participants can actively participate.

[0564] "Speech recognition means" refers to technology that recognizes speech in real time and converts it into text data.

[0565] A "key point extraction method" is a technology that uses natural language processing to extract important information from text data and automatically generate meeting minutes.

[0566] "Meeting minutes distribution method" refers to the technology for transmitting generated meeting minutes to a communication device.

[0567] A "task management method" is a technology that automatically lists and manages tasks that arise during a meeting.

[0568] A "scheduling adjustment method" is a technology that adjusts meeting schedules based on information about dates and provides necessary information immediately.

[0569] "Methods for analyzing emotions" refer to technologies that analyze participants' emotions from data such as audio and video.

[0570] "Methods for dynamically adjusting the progress of a conversation" refer to techniques that adjust the way a conversation progresses in a timely manner based on analyzed emotions.

[0571] "Methods for providing information tailored to participants' emotions" refers to techniques for providing appropriate information according to the emotional state of the participants.

[0572] This invention is implemented through a system that combines speech recognition technology, natural language processing technology, and sentiment analysis technology. Specifically, a terminal captures audio and video in real time as soon as a meeting begins. The terminal sends the captured data to a server. The server uses a speech recognition library to convert the audio into text data. In this case, "speech_recognition" is used as a specific example. Important information is extracted from this text data using natural language processing technology. "NLPProcessor" is used for this technology.

[0573] The server analyzes the generated text data and uses an emotion engine to analyze participants' emotions. This process employs "EmotionRecognizer." Based on the results, meeting minutes are automatically generated and distributed in real time. Furthermore, the conversation is dynamically adjusted based on participants' emotions, and information is provided as needed. Task management technology is also applied to list and manage tasks that arise during the meeting.

[0574] A concrete example of its use is in a workplace project meeting. If the emotion engine detects a decline in participants' interest, the server can immediately provide additional materials or explanations to rekindle their interest. This ensures an environment where everyone remains actively engaged in the meeting.

[0575] By using a generative AI model to provide information tailored to the participant's emotions, the following prompt can be used: "If a participant is perceived to be showing anxiety during a discussion about travel plans, what information should be provided?"

[0576] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0577] Step 1:

[0578] The terminal captures audio and video data in real time using its microphone and camera as soon as the meeting begins. It acquires audio and video data as input and converts them into a digital format. This data is then sent to the server for further processing.

[0579] Step 2:

[0580] The server converts received audio data into text data using the "speech_recognition" library. It receives an audio file as input and formats it into a string in real time using speech recognition technology. The output, as text data, is used for subsequent natural language processing.

[0581] Step 3:

[0582] The server analyzes the converted text data using an "NLP Processor" to extract important information. It receives text data as input and applies natural language processing algorithms to identify key points. The output is key information for use in generating meeting minutes.

[0583] Step 4:

[0584] The server automatically generates meeting minutes from the key information obtained above and distributes them to the communication device. It receives key information as input, applies an automatic format, and creates formatted meeting minutes. The output is meeting minutes data accessible to participants.

[0585] Step 5:

[0586] The server simultaneously analyzes participants' emotions using the "EmotionRecognizer" emotion engine based on video data. The input is video data, and emotions are analyzed through facial recognition and voice tone analysis. The output is information about the participants' emotional state.

[0587] Step 6:

[0588] The server dynamically adjusts the flow of the conversation based on emotional information and selects information and additional explanations to provide to participants. It takes emotional states and meeting minutes as input and determines the content and timing of information provision accordingly. The output is the selected information and any additional materials presented.

[0589] Step 7:

[0590] Users review meeting minutes and additional information delivered from the server via their terminals and take necessary actions. The input consists of delivered data, allowing users to make decisions and execute subsequent steps and tasks based on their own judgment.

[0591] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0592] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0593] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0594] [Fourth Embodiment]

[0595] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0596] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0597] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0598] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0599] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0600] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0601] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0602] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0603] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0604] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0605] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0606] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0607] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0608] This invention is a system that improves the efficiency of meetings by integrating speech recognition technology and natural language processing technology. Users start a meeting using a meeting terminal. The terminal captures the audio during the meeting in real time via a microphone and transmits the audio data to a server.

[0609] The server quickly converts audio data into text data using a speech recognition module. This converted text data is then processed by a natural language processing module on the server to extract key points and generate an outline of the meeting minutes. The meeting minutes are automatically generated by the server and immediately delivered to the terminal. Users can review the minutes on their terminal and add comments as needed.

[0610] Furthermore, the server automatically lists any new tasks that arise during the meeting. Each task is accompanied by information such as a deadline and the person responsible, and is managed within the server. Users can check this task information from their terminals and manage its progress.

[0611] Furthermore, the server adjusts meeting schedules based on the user's schedule information. It also instantly searches for relevant documents and data as needed and provides them to the user's terminal. This ensures that users can access information without delay during meetings and focus on the discussion.

[0612] As a concrete example, consider a user holding a monthly reporting meeting. The user's device connects to the server at the start of the meeting and transmits what is said during the meeting to the server in real time. The server transcribes the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion about cost reduction takes place, the content is summarized and immediately shared with all participants.

[0613] This invention significantly improves meeting productivity and provides a system that helps users concentrate on the discussion.

[0614] The following describes the processing flow.

[0615] Step 1:

[0616] The terminal captures audio via its microphone at the start of the meeting. The captured audio data is divided into packets in real time and sent to the server.

[0617] Step 2:

[0618] The server inputs the received audio packets into the speech recognition module, converting the audio into text data. This allows the audio information to be treated as text information.

[0619] Step 3:

[0620] The server sends the text data to a natural language processing module to extract important information and key points from the meeting. The extracted information is temporarily stored on the server.

[0621] Step 4:

[0622] The server automatically creates meeting minutes based on the extracted key points. The created meeting minutes data is then sent back to the terminal.

[0623] Step 5:

[0624] Users can review meeting minutes received on their devices and add comments or corrections as needed. The minutes can also be saved and shared with other participants.

[0625] Step 6:

[0626] The server automatically creates a list of tasks discussed during the meeting and adds deadlines and assignees to each task. The generated task list is then delivered to the terminals.

[0627] Step 7:

[0628] Users can view their task list on their device and manage their progress. The completion status of each task is synchronized with the server, making it easy to track overall progress.

[0629] Step 8:

[0630] The server integrates with the user's calendar data to schedule the next meeting. It also displays necessary documents and information on the user's device during the meeting, improving meeting efficiency.

[0631] (Example 1)

[0632] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0633] Modern meetings are becoming increasingly complex, and with the growing volume and processing speed of information, efficient information management and record-keeping are required to allow participants to focus on the discussion. However, manual creation of meeting minutes, task management, and provision of related materials are time-consuming and labor-intensive, potentially leading to decreased productivity. Therefore, there is a need for a means to properly manage meeting information in real time and to efficiently advance discussions.

[0634] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0635] In this invention, the server includes speech recognition means, key point extraction means, and meeting minute distribution means. This makes it possible to convert audio during a meeting into text data in real time, automatically extract key points of important discussions to generate meeting minutes, and immediately distribute them to participants. This promotes the immediate sharing of information and helps participants concentrate on the discussion.

[0636] "Speech recognition means" refers to a technology or device for converting speech data into text data in real time.

[0637] A "key point extraction method" is a technique or device that uses natural language processing to extract important information or key points of discussion from text data.

[0638] "Meeting minutes distribution means" refers to a technology or device that distributes generated meeting minutes to information terminals via communication.

[0639] "Work item management means" refers to a technology or device for automatically listing work items that arise during a meeting and for managing those work items.

[0640] "Schedule adjustment means" refers to technology or equipment for adjusting meeting schedules based on time information and providing necessary information immediately.

[0641] "Means of providing information" refers to a technology or device that provides information related to an ongoing meeting in real time based on text data acquired by speech recognition means.

[0642] "Progress management means" refers to a technology or device for identifying deadlines and responsible persons for work items extracted by work item management means, and for individually managing their progress.

[0643] This invention is a system that combines speech recognition technology and natural language processing technology, designed to streamline meetings. Users initiate a meeting using a conference terminal. The terminal uses a high-sensitivity microphone to capture audio in real time during the meeting. The audio data is stored digitally and transmitted to a server via a secure protocol (e.g., TLS / SSL).

[0644] The server uses a speech recognition module (for example, Google Speech-to-Text API, a common speech recognition API) to quickly convert audio data into text data. Then, a natural language processing module (e.g., SpaCy or NLTK) is used to analyze the converted text data and extract the key points of the meeting. The automatically generated meeting minutes are then immediately delivered to terminals by the server.

[0645] Users can view meeting minutes on their devices and add comments as needed. In this process, the server has a function to list new tasks that arise during the meeting. These tasks are accompanied by deadlines and assigned personnel information generated by an AI model, and are viewed and managed through the device. Furthermore, the server adjusts meeting schedules based on the user's time information and instantly searches for and provides necessary materials and data. Therefore, users can access necessary information without delay, even during meetings, enabling efficient discussions.

[0646] As a concrete example, consider a scenario where a user holds a regular progress report meeting. In this case, the user's device connects to the server at the start of the meeting and transmits audio to the server in real time. The server converts the audio into text, automatically extracts important reports and decisions, and creates meeting minutes. For example, if a discussion takes place regarding the efficiency of management resources, the content is summarized and immediately shared with all participants.

[0647] An example of a prompt message would be, "Please extract the key points from the following audio and create meeting minutes." This would significantly improve meeting productivity and provide users with an environment where they can focus on the discussion.

[0648] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0649] Step 1:

[0650] Users initiate a meeting using a conference terminal. The terminal captures the meeting audio in real time using its built-in high-sensitivity microphone. The input is the audio from the meeting, and the output is a digital audio file (e.g., WAV format). The terminal temporarily saves the audio data to a PC or online storage to prevent data loss.

[0651] Step 2:

[0652] The terminal sends the stored audio data to the server using a secure protocol (e.g., TLS / SSL). This process involves data encryption. The input is the captured audio data, and the output is the encrypted audio data. This ensures secure data transfer.

[0653] Step 3:

[0654] The server converts received audio data into text data using a speech recognition module (e.g., a common speech recognition API). The input is the received audio data, and the output is text data (e.g., in text format). The server analyzes the audio waveform data, recognizes phonemes and words, and performs this conversion.

[0655] Step 4:

[0656] The server analyzes the generated text data using a natural language processing module (e.g., SpaCy) to extract the key points of the meeting. The input is the converted text data, and the output is a summary of the key points (e.g., summary text). The server calculates keyword frequency and sentence weighting within the text to extract important information.

[0657] Step 5:

[0658] The server automatically generates meeting minutes based on extracted key points. The input is data summarizing the key points, and the output is a formatted meeting minute (e.g., PDF or DOC format). This generation process utilizes a template engine and style processing. The server then distributes the generated meeting minutes to information terminals.

[0659] Step 6:

[0660] The server automatically lists work items that arise during a meeting using a generation AI model. The input is text data of the meeting content, and the output is a list of work items. The server analyzes the command structure and task-related expressions within the text to identify the items.

[0661] Step 7:

[0662] Users can review meeting minutes and work items delivered via their terminals and add comments and corrections. Inputs are the delivered meeting minutes and work items, while outputs are supplementary meeting minutes and an updated work item list. Users can manage their progress through their terminals.

[0663] (Application Example 1)

[0664] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0665] In emergency situations, there is a problem in quickly and accurately gathering on-site audio information and transmitting it to the command center and response team with speed and accuracy. In particular, it is necessary to efficiently assess the importance of information generated on-site, quickly extract the necessary information, and share it.

[0666] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0667] In this invention, the server includes an acoustic recognition means, an information extraction means, and an information aggregation means. This makes it possible to quickly convert voice information into text, automatically extract important information, and immediately provide a report to the command center at the scene of an emergency response.

[0668] "Acoustic recognition means" refers to a means of acquiring sound as a digital signal and converting it into language data in real time.

[0669] An "information extraction method" is a means of extracting important features and keywords from acquired language data using natural language processing and automatically generating records.

[0670] A "record distribution method" is a means of distributing generated records to communication devices and sharing them with relevant parties.

[0671] A "task management system" is a means of automatically listing and managing tasks that arise during meetings or interactions.

[0672] A "schedule adjustment method" is a means of adjusting meeting schedules based on date information and providing necessary information immediately.

[0673] An "information aggregation method" is a means of aggregating on-site audio information, extracting important keywords, and generating an automated report in situations requiring action.

[0674] One embodiment of this invention consists of a system integrating acoustic recognition means, information extraction means, recording and distribution means, and the like. This system uses specific hardware and software to acquire sound in real time and convert it into language data.

[0675] Specifically, a device such as a smartphone captures audio data with its microphone and converts it into language data using the Google Cloud Speech-to-Text API. This speech recognition requires an acoustic model using TensorFlow.

[0676] The converted language data is sent to a server, where the SpaCy natural language processing library is used by an information extraction system to identify key keywords and automatically generate records. The generated records are immediately distributed to relevant parties using cloud communication capabilities. The information aggregation system quickly processes information that is particularly needed in emergencies and provides important content to command centers and other relevant locations.

[0677] In the operation of this system, the server lists tasks as they arise and provides a task management function that makes it easy for users to check their progress from their terminals. In addition, the scheduling mechanism allows for quick and efficient management of meeting dates, minimizing inconvenience caused by schedule changes.

[0678] For example, if keywords requiring emergency response, such as "fire," "smoke," and "evacuation," are identified at a fire response site, a report containing these keywords will be immediately generated and distributed. This enables rapid information sharing between the field and the command center.

[0679] An example of a prompt for the generating AI model would be the text: "Please tell me how to extract important keywords from audio data at an emergency response site and automatically generate a report to the command center immediately."

[0680] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0681] Step 1:

[0682] The device uses a microphone to capture audio data from the field and generates a digitized audio signal. This signal is sent to the Google Cloud Speech-to-Text API. The input is analog audio, and the output is a digitized audio signal.

[0683] Step 2:

[0684] The server uses the Google Cloud Speech-to-Text API to analyze the speech signal and convert it into language data in real time. An acoustic model is used for this conversion, sequentially outputting text data from the speech data. The input is a digitized speech signal, and the output is text data.

[0685] Step 3:

[0686] The server uses SpaCy to extract important keywords from text data. Natural language processing is employed here to clarify key features from the linguistic data. The input is text data, and the output consists of extracted keywords and important content.

[0687] Step 4:

[0688] As a means of information aggregation, the server instantly generates important reports based on extracted keywords. These reports contain all the necessary content and are quickly transmitted to the command center. The input is the important information, and the output is the generated report.

[0689] Step 5:

[0690] Users view reports generated on their devices and report the situation to the command center as needed. User review of the reports facilitates smoother communication. The input is the generated reports, and the output is the user's actions and feedback.

[0691] Through this process, we will utilize generative AI models to enable immediate processing and transmission of information at emergency sites.

[0692] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0693] This invention is a system that improves meeting efficiency and participant engagement by incorporating an emotion engine that recognizes user emotions, in addition to speech recognition technology and natural language processing technology. The system is composed of three main elements: a server, terminals, and users.

[0694] When a user starts a meeting, their device uses its microphone and camera to capture audio and video data in real time. The device sends the captured data to a server. The server uses a speech recognition module to convert the audio into text data, and simultaneously analyzes the user's facial expressions and tone of voice from the video data to recognize their emotions.

[0695] The server uses a natural language processing module to extract important information from text data, summarize the key points of the meeting, and generate meeting minutes. Simultaneously, an emotion engine identifies the user's emotional state, and the meeting's progress can be dynamically adjusted based on this information. For example, if a participant is deemed confused, an alert will be displayed on the terminal to provide additional explanations or relevant materials.

[0696] Users can view meeting minutes generated on their devices, and tasks that arise during the meeting are displayed in real time. The server assists in optimizing the meeting's outcome based on user sentiment information. This creates an environment where discussions are not one-sided and all participants can actively engage.

[0697] As a concrete example, when holding a strategy meeting, the user activates a terminal to start the meeting. The terminal captures audio and video, and the server performs analysis. If a presentation is given during the meeting and the emotion engine detects a decline in participants' interest or understanding, the server suggests options to readjust the focus of the meeting based on that information. In this way, this invention functions not only as a means of transmitting information, but also as a tool to improve the quality of communication.

[0698] The following describes the processing flow.

[0699] Step 1:

[0700] The device captures audio and video using its built-in microphone and camera as soon as the meeting starts. The captured data is sent to the server in real time.

[0701] Step 2:

[0702] The server converts the audio data received by the speech recognition module into text data. This process transcribes the audio content into text.

[0703] Step 3:

[0704] The server simultaneously analyzes the video data using an emotion engine, recognizing the user's emotional state from their facial expressions, facial movements, and voice tone. Changes in emotion and specific states are identified.

[0705] Step 4:

[0706] The server analyzes text data through a natural language processing module and extracts important information and key points from the meeting. Based on the extraction results, meeting minutes are automatically generated.

[0707] Step 5:

[0708] Based on the emotional information detected by the emotion engine, the server analyzes the user's level of understanding and reactions, and provides suggestions for meeting progress or additional materials if necessary. This information is then notified to the terminal.

[0709] Step 6:

[0710] Users can review the meeting minutes and suggested supplementary information generated on their devices, allowing them to gain a deeper understanding of the meeting content and participate in the discussion.

[0711] Step 7:

[0712] The server automatically lists any new tasks that arise during the meeting, identifying deadlines and assigning responsibilities. The generated task list is also delivered to the user's terminal, allowing them to monitor progress.

[0713] Step 8:

[0714] Based on user feedback, the server prepares post-meeting feedback and provides it to users via their terminals. This helps improve future meetings.

[0715] (Example 2)

[0716] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0717] Traditional meeting systems posed a significant challenge in creating meeting minutes by recording participants' speeches. This involved tedious tasks such as transcribing audio data into text and extracting key points, and it was difficult to adjust the meeting's pace to accommodate participants' emotional states and levels of understanding. As a result, meetings could become one-sided, potentially leading to misunderstandings or decreased motivation among some participants. This situation highlights the need for technology that can significantly improve meeting efficiency and participant engagement.

[0718] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0719] In this invention, the server includes a speech recognition means that recognizes speech in real time and converts it into text data, a key point extraction means that extracts key points from the text data using natural language processing and automatically generates meeting minutes, and an emotion recognition means that analyzes the facial expressions and voice tone of participants during the meeting to recognize their emotions. This makes it possible to improve the efficiency of meetings and provide an environment in which all participants can actively engage.

[0720] "Speech recognition means" refers to a function that processes speech data into text data in real time.

[0721] A "key point extraction method" is a function that extracts important information from text data and automatically compiles it into meeting minutes or reports.

[0722] "Emotion recognition means" refers to a function that analyzes participants' facial expressions and tone of voice to identify their emotional state.

[0723] A "meeting minutes distribution method" is a function that sends the generated meeting minutes to another device, making them accessible to all participants.

[0724] A "task management system" is a function for listing tasks that arise during a meeting and managing their progress.

[0725] A "meeting management tool" is a function that dynamically adjusts the progress of a meeting according to the participants' emotions and level of understanding, thereby promoting smooth communication.

[0726] A "schedule adjustment tool" is a function that optimizes meeting schedules based on date information and provides necessary information immediately.

[0727] "Means of providing information" refers to a function that provides timely reference materials related to the ongoing meeting, thereby supporting participants' understanding.

[0728] A "progress management method" is a function that allows managers to individually check the progress of selected tasks by specifying deadlines and responsible persons.

[0729] This system provides a program that integrates speech recognition, natural language processing, and emotion recognition technologies to improve meeting efficiency and promote active participant engagement. The system primarily consists of server, terminal, and user elements.

[0730] First, when a user starts a meeting, the device captures audio and video data in real time using its built-in microphone and camera. The device immediately sends this data to the server. To maintain secure communication, the device is required to use security protocols such as HTTPS for data transfer.

[0731] Next, the server uses speech recognition to convert the audio data into text data. This process employs common speech recognition technologies; for example, a cloud-based speech recognition API can be used. The server also uses emotion recognition on the video data to identify participants' emotions from their facial expressions and tone of voice.

[0732] The server applies natural language processing techniques to the converted text data, extracting key points and automatically generating meeting minutes. This process utilizes widely available natural language processing APIs. In parallel, it can also adjust the meeting's progress based on sentiment recognition; for example, if it analyzes that participants are confused, it can provide additional explanations or supplementary materials.

[0733] Users can view meeting minutes generated in real time and tasks performed during the meeting through their devices. This allows all participants to understand the meeting content and obtain necessary information in a timely manner.

[0734] As a concrete example, consider a scenario where a user initiates a corporate strategy meeting, and their device transmits audio and video to a server. Based on the information received, the server evaluates the level of understanding of all participants and, if interest wanes, suggests readjusting the flow of the meeting. In this way, the quality of communication is improved, making the meeting more effective.

[0735] When using a generative AI model, you can input text-based instructions as an example of a prompt, such as "Based on the participant sentiment analysis results, please formulate the agenda for the next meeting."

[0736] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0737] Step 1:

[0738] When a user starts a meeting, the device activates its microphone and camera and captures audio and video data in real time. The input consists of the voices and facial expressions of the meeting participants. The device uses digital signal processing to transform the analog data and outputs it as digital audio and video data.

[0739] Step 2:

[0740] The terminal sends the captured digital audio and video data to the server. The input is the digital data generated in step 1. For output, secure data transfer via protocols such as HTTPS takes place, and the data reaches the server.

[0741] Step 3:

[0742] The server applies a speech recognition module to the received audio data, converting the audio into text data. The input is digital audio data sent from the terminal. Speech recognition technology is used to process the data and generate text data as output.

[0743] Step 4:

[0744] The server analyzes the video data and identifies emotions using the participants' facial expressions and voice tone. The input is the digital video data transmitted in step 2. Facial feature points are extracted, and a machine learning algorithm is used to obtain an estimated result of the emotional state as output.

[0745] Step 5:

[0746] The server applies natural language processing to the converted text data, summarizing key information and automatically generating meeting minutes. The input is the text data obtained in step 3. It performs data processing using keyword extraction and summarization algorithms to generate meeting minutes as output.

[0747] Step 6:

[0748] The server dynamically adjusts the meeting's progress using the results of emotion recognition. The input is the emotional state data obtained in step 4. If a participant is determined to be confused, adjustments are made to the meeting, such as presenting additional materials. The output is fed back to the terminal as an optimized meeting plan and alert information.

[0749] Step 7:

[0750] Users can view meeting minutes and task lists generated through their terminals and track their work in real time as needed. Inputs consist of meeting minutes data and task data generated during the meeting, sent from the server to the terminal. Outputs are documented information displayed in the user interface, making it easier for users to understand the meeting content.

[0751] (Application Example 2)

[0752] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0753] Conventional meeting systems often fail to consider participants' emotional states when providing information or conducting meetings, leading to decreased understanding and engagement. Furthermore, insufficient automated support for meeting efficiency often resulted in one-sided discussions, making it difficult to create an environment where all participants could actively engage. This invention aims to solve these problems and realize communication support that reflects participants' emotions.

[0754] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0755] In this invention, the server includes: speech recognition means for recognizing speech in real time and converting it into text data; key point extraction means for extracting key points from the text data using natural language processing and automatically generating meeting minutes; meeting minute distribution means for distributing the generated meeting minutes to a communication device; task management means for automatically listing tasks that arise during a meeting and managing those tasks; schedule adjustment means for adjusting the meeting schedule based on date information and providing necessary information immediately; emotions analysis means; dynamic adjustment of the conversation based on the analyzed emotions; and means for providing information in accordance with the emotions of the participants. This makes it possible to conduct a meeting while considering the emotional state of the participants and to provide an efficient meeting environment in which all participants can actively participate.

[0756] "Speech recognition means" refers to technology that recognizes speech in real time and converts it into text data.

[0757] A "key point extraction method" is a technology that uses natural language processing to extract important information from text data and automatically generate meeting minutes.

[0758] "Meeting minutes distribution method" refers to the technology for transmitting generated meeting minutes to a communication device.

[0759] A "task management method" is a technology that automatically lists and manages tasks that arise during a meeting.

[0760] A "scheduling adjustment method" is a technology that adjusts meeting schedules based on information about dates and provides necessary information immediately.

[0761] "Methods for analyzing emotions" refer to technologies that analyze participants' emotions from data such as audio and video.

[0762] "Methods for dynamically adjusting the progress of a conversation" refer to techniques that adjust the way a conversation progresses in a timely manner based on analyzed emotions.

[0763] "Methods for providing information tailored to participants' emotions" refers to techniques for providing appropriate information according to the emotional state of the participants.

[0764] This invention is implemented through a system that combines speech recognition technology, natural language processing technology, and sentiment analysis technology. Specifically, a terminal captures audio and video in real time as soon as a meeting begins. The terminal sends the captured data to a server. The server uses a speech recognition library to convert the audio into text data. In this case, "speech_recognition" is used as a specific example. Important information is extracted from this text data using natural language processing technology. "NLPProcessor" is used for this technology.

[0765] The server analyzes the generated text data and uses an emotion engine to analyze participants' emotions. This process employs "EmotionRecognizer." Based on the results, meeting minutes are automatically generated and distributed in real time. Furthermore, the conversation is dynamically adjusted based on participants' emotions, and information is provided as needed. Task management technology is also applied to list and manage tasks that arise during the meeting.

[0766] A concrete example of its use is in a workplace project meeting. If the emotion engine detects a decline in participants' interest, the server can immediately provide additional materials or explanations to rekindle their interest. This ensures an environment where everyone remains actively engaged in the meeting.

[0767] By using a generative AI model to provide information tailored to the participant's emotions, the following prompt can be used: "If a participant is perceived to be showing anxiety during a discussion about travel plans, what information should be provided?"

[0768] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0769] Step 1:

[0770] The terminal captures audio and video data in real time using its microphone and camera as soon as the meeting begins. It acquires audio and video data as input and converts them into a digital format. This data is then sent to the server for further processing.

[0771] Step 2:

[0772] The server converts received audio data into text data using the "speech_recognition" library. It receives an audio file as input and formats it into a string in real time using speech recognition technology. The output, as text data, is used for subsequent natural language processing.

[0773] Step 3:

[0774] The server analyzes the converted text data using an "NLP Processor" to extract important information. It receives text data as input and applies natural language processing algorithms to identify key points. The output is key information for use in generating meeting minutes.

[0775] Step 4:

[0776] The server automatically generates meeting minutes from the key information obtained above and distributes them to the communication device. It receives key information as input, applies an automatic format, and creates formatted meeting minutes. The output is meeting minutes data accessible to participants.

[0777] Step 5:

[0778] The server simultaneously analyzes participants' emotions using the "EmotionRecognizer" emotion engine based on video data. The input is video data, and emotions are analyzed through facial recognition and voice tone analysis. The output is information about the participants' emotional state.

[0779] Step 6:

[0780] The server dynamically adjusts the flow of the conversation based on emotional information and selects information and additional explanations to provide to participants. It takes emotional states and meeting minutes as input and determines the content and timing of information provision accordingly. The output is the selected information and any additional materials presented.

[0781] Step 7:

[0782] Users review meeting minutes and additional information delivered from the server via their terminals and take necessary actions. The input consists of delivered data, allowing users to make decisions and execute subsequent steps and tasks based on their own judgment.

[0783] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0784] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0785] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0786] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0787] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0788] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0789] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0790] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0791] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0792] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0793] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0794] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0795] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0796] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0797] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0798] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0799] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0800] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0801] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0802] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0803] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0804] The following is further disclosed regarding the embodiments described above.

[0805] (Claim 1)

[0806] A speech recognition method that recognizes speech in real time and converts it into text data,

[0807] A key point extraction method that uses natural language processing to extract key points from text data and automatically generate meeting minutes,

[0808] A meeting minutes distribution means for distributing the generated meeting minutes to a communication device,

[0809] A task management system that automatically lists tasks that arise during meetings and manages those tasks,

[0810] A system that includes a scheduling mechanism that adjusts meeting schedules based on scheduling information and provides necessary information immediately.

[0811] (Claim 2)

[0812] The system according to claim 1, further comprising a material provision means that provides materials related to an ongoing meeting in real time based on text data acquired by a speech recognition means.

[0813] (Claim 3)

[0814] The system according to claim 1, further comprising a progress management means for identifying deadlines and responsible persons for tasks extracted by the task management means and for individually managing the progress of those tasks.

[0815] "Example 1"

[0816] (Claim 1)

[0817] A speech recognition method that recognizes speech in real time and converts it into text data,

[0818] A key-point extraction method that uses natural language processing to extract key points from text data and automatically generates meeting minutes,

[0819] A meeting minutes distribution method that distributes the generated meeting minutes to information terminals,

[0820] A work item management system that automatically lists the work items that arise during a meeting and manages those work items,

[0821] A system that includes a scheduling mechanism that adjusts meeting schedules based on time information and provides necessary information immediately.

[0822] (Claim 2)

[0823] The system according to claim 1, further comprising a material provision means that provides materials related to an ongoing meeting in real time based on text data acquired by a speech recognition means.

[0824] (Claim 3)

[0825] The system according to claim 1, further comprising a progress management means for identifying deadlines and responsible persons for work items extracted by a work item management means, and for individually managing the progress of each work item.

[0826] "Application Example 1"

[0827] (Claim 1)

[0828] A sound recognition means that recognizes speech in real time and converts it into language data,

[0829] An information extraction method that uses natural language processing to extract important features from language data and automatically generates records,

[0830] A record distribution means for distributing the generated records to communication devices,

[0831] A task management system that automatically lists and manages tasks that arise during a meeting,

[0832] A scheduling tool that adjusts meeting schedules based on date information and provides necessary information immediately,

[0833] An information aggregation means that quickly gathers on-site information in situations requiring action, extracts important keywords, and automatically generates reports.

[0834] ...

[0835] A system that includes this.

[0836] (Claim 2)

[0837] The system according to claim 1, further comprising an information providing means that provides information relating to an ongoing meeting in real time based on language data acquired by a speech recognition means.

[0838] (Claim 3)

[0839] The system according to claim 1, further comprising a progress management means for identifying deadlines and responsible persons for tasks extracted by the work management means and for individually managing the progress of those tasks.

[0840] "Example 2 of combining an emotion engine"

[0841] (Claim 1)

[0842] A speech recognition method that recognizes speech in real time and converts it into text data,

[0843] A key point extraction method that uses natural language processing to extract key points from text data and automatically generate meeting minutes,

[0844] An emotion recognition method that analyzes the facial expressions and voice tone of participants during a meeting to recognize their emotions,

[0845] A meeting minutes distribution means that distributes the generated meeting minutes to an information transmission device,

[0846] A task management system that automatically lists and manages tasks that arise during meetings,

[0847] A progress adjustment means that dynamically adjusts the progress of a meeting based on emotional information acquired by an emotion recognition means,

[0848] A system that includes a scheduling mechanism that adjusts meeting schedules based on scheduling information and provides necessary information immediately.

[0849] (Claim 2)

[0850] The system according to claim 1, further comprising a material provision means that provides reference materials related to an ongoing meeting in real time based on text data acquired by a speech recognition means.

[0851] (Claim 3)

[0852] The system according to claim 1, further comprising a progress management means for identifying deadlines and responsible persons for tasks extracted by the work management means and for individually managing the progress of those tasks.

[0853] "Application example 2 when combining with an emotional engine"

[0854] (Claim 1)

[0855] A speech recognition method that recognizes speech in real time and converts it into text data,

[0856] A key point extraction method that uses natural language processing to extract key points from text data and automatically generate meeting minutes,

[0857] A meeting minutes distribution means for distributing the generated meeting minutes to a communication device,

[0858] A task management system that automatically lists tasks that arise during meetings and manages those tasks,

[0859] A scheduling tool that adjusts meeting schedules based on scheduling information and provides necessary information immediately,

[0860] A means of analyzing emotions,

[0861] A means of dynamically adjusting the progress of a conversation based on analyzed emotions,

[0862] A means of providing information tailored to the participants' emotions,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, further comprising a material provision means that provides materials related to an ongoing meeting in real time based on text data acquired by a speech recognition means.

[0866] (Claim 3)

[0867] The system according to claim 1, further comprising a progress management means for identifying deadlines and responsible persons for tasks extracted by the task management means and for individually managing the progress of those tasks. [Explanation of Symbols]

[0868] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A sound recognition means that recognizes speech in real time and converts it into language data, An information extraction method that uses natural language processing to extract important features from language data and automatically generates records, A record distribution means for distributing the generated records to communication devices, A task management system that automatically lists and manages tasks that arise during a meeting, A scheduling tool that adjusts meeting schedules based on date information and provides necessary information immediately, An information aggregation means that quickly gathers on-site information in situations requiring action, extracts important keywords, and automatically generates reports. A system that includes this.

2. The system according to claim 1, further comprising an information providing means that provides information relating to an ongoing meeting in real time based on language data acquired by a speech recognition means.

3. The system according to claim 1, further comprising a progress management means for identifying deadlines and responsible persons for tasks extracted by the work management means and for individually managing the progress of those tasks.

Images