System

Abstract

To provide a system that extracts information across multiple conference information.SOLUTION: A system includes means for: receiving an inquiry from a user; extracting information related to the inquiry from video data or voice data representing multiple conferences or conversations, using a generative AI model and a prompt sentence instructing the extraction of information related to the inquiry from the video data or voice data representing the multiple conferences or conversations; and providing the extracted information related to the inquiry to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, communication tools and emails offer search functions, but with conferencing tools like Zoom, even if recordings are available, searching is difficult, let alone cross-searching multiple meetings. In particular, when you find yourself thinking, "I remember talking about this before, but what did that person say? When and in which meeting did I hear it...?", extracting relevant content from numerous past meeting information (video and audio recordings) is time-consuming and inefficient. [Means for solving the problem]

[0005] The present invention provides a means for receiving multiple conference information (video and audio recordings) as input, extracting information related to specific content, and presenting the extracted information to the user. It also includes a means for extracting information based on a user's inquiry and a means for extracting information across multiple conference information. This allows users to efficiently obtain the information they need from past conference information. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The system of the present invention has a means for receiving multiple pieces of conference information (video and audio recordings) as input. Specifically, a user inputs recording data of a conference held using a conference tool (e.g., Zoom) into the system.

[0029] "Example 2"

[0030] The system of the present invention has a means for extracting information related to specific content. Specifically, when a user makes a query such as "What did that person say?", the system extracts related content from the input conference information. This extraction is performed using voice recognition technology and natural language processing technology.

[0031] "Example 3"

[0032] The system of the present invention has a means for presenting the extracted information to the user. Specifically, the system presents the extracted information to the user in text format. The system can also present the date, time, participants, and other information of related meetings.

[0033] "Example 4"

[0034] The system of the present invention has a means for extracting information based on a user's inquiry. Specifically, when a user inquires about a specific keyword or phrase, the system extracts information related to that keyword or phrase from the conference information.

[0035] "Example 5"

[0036] The system of the present invention has a means for extracting information across multiple conference records. Specifically, when a user inquires about discussions that took place in multiple conferences, the system extracts related information from the multiple conference records. This allows the user to obtain information across multiple conferences at once, rather than just one conference.

[0037] The processing flow of each embodiment will be described below.

[0038] "Example 1"

[0039] Step 1: Users input recording data of meetings they have held using a conferencing tool (e.g., Zoom) into the system.

[0040] "Example 2"

[0041] Step 1: The user asks a question such as, "What did that person say?"

[0042] Step 2: The system extracts relevant content from the input meeting information using voice recognition and natural language processing technologies.

[0043] "Example 3"

[0044] Step 1: The system presents the extracted information to the user in text format.

[0045] Step 2: Provide relevant meeting dates, times, participants, and other information.

[0046] "Example 4"

[0047] Step 1: User queries for a specific keyword or phrase.

[0048] Step 2: The system extracts information related to the keyword or phrase from the meeting information.

[0049] "Example 5"

[0050] Step 1: A user inquires about discussions that took place across multiple meetings.

[0051] Step 2: The system extracts relevant information from multiple conferences, allowing users to obtain information across multiple conferences at once, rather than just a single conference.

[0052] Example 1

[0053] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0054] Conventional meeting information management systems have had difficulty efficiently extracting necessary information from recorded meeting data and generating summaries. Furthermore, they lacked a means for users to quickly grasp the content of a meeting. This resulted in the problem of requiring a great deal of time and effort when reviewing the content of a meeting later.

[0055] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0056] In this invention, the server includes a means for receiving multiple pieces of conference information (video and audio recordings) as input and converting the audio data into text, a means for inputting a prompt sentence into a generative AI model to generate a summary, and a means for presenting the generated summary to the user. This allows for efficient generation of summaries from recorded conference data, enabling users to quickly understand the content of the conference.

[0057] "Meeting information" refers to video and audio data that records the contents of the meeting.

[0058] "Means for converting voice data to text" refers to technology or devices for analyzing voice data and generating corresponding text data.

[0059] A "generative AI model" refers to an artificial intelligence model that performs natural language processing based on input data to generate summaries and other text.

[0060] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0061] "Means for generating a summary" refers to technology or equipment for creating a summary from input text data using a generative AI model.

[0062] "User presentation means" refers to the techniques and devices used to display or provide the generated summary to the user.

[0063] This invention begins when a user uploads a video file of a conference recorded with a conference tool to the system. The user logs in to the system's web portal, selects the recorded data, and clicks the upload button. The server receives the conference recording data uploaded by the user and temporarily stores it.

[0064] The server then converts the audio portion of the recording into text using the Google® Cloud Speech-to-Text API. Specifically, the server extracts the audio track from the recording and sends it to the API. The API parses the audio data and returns the corresponding text data.

[0065] The server then inputs a prompt to the generative AI model (e.g., OpenAI's GPT-4). The prompt includes the converted text data and instructions for generating a summary. For example, the prompt might read, "The following text is the content of a one-hour meeting. Please write a summary of this meeting in 500 characters or less."

[0066] The generative AI model generates a summary of the meeting based on this prompt. The server receives the summary and provides it to the user, who can review it through the system's interface. The server then associates the summary with the user's account and saves it for later access.

[0067] As a concrete example, consider the case where a user uploads a recording of a one-hour Zoom meeting to the system. The server first converts the hour's worth of audio data into text using the Google Cloud Speech-to-Text API. Then, it inputs the following prompt to the generative AI model (GPT-4):

[0068] "The text below is from a one-hour meeting. Please write a summary of this meeting in 500 words or less."

[0069] The generative AI model generates a summary of the meeting based on this prompt, and the server provides the generated summary to the user, who can then view it through the system interface.

[0070] The above is a specific embodiment for carrying out the present invention. This system makes it possible to efficiently generate summaries from recorded data of a conference, enabling users to quickly grasp the contents of the conference.

[0071] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0072] Step 1:

[0073] A user uploads a meeting recording.

[0074] A user logs in to the system's web portal, selects the recording data, and clicks the upload button. The input is the meeting recording data selected by the user (e.g., meeting_recording.mp4), and the output is the recording data sent to the server.

[0075] Step 2:

[0076] The server receives the recording data.

[0077] The server receives the conference recording data uploaded by the user and temporarily stores it. The input is the recording data sent by the user, and the output is the recording data saved in the server's storage.

[0078] Step 3:

[0079] The server converts the audio data into text.

[0080] The server converts the audio portion of the recording into text using the Google Cloud Speech-to-Text API. Specifically, the server extracts the audio track from the recording and sends it to the API. The input is the audio track of the recording, and the output is the text data returned by the API.

[0081] Step 4:

[0082] The server inputs a prompt sentence into the generative AI model.

[0083] The server inputs a prompt to the generative AI model (e.g., OpenAI's GPT-4). The prompt includes the converted text data and instructions to generate a summary. The input is the text data and the prompt, and the output is the summary returned by the generative AI model. For example, the prompt might be, "The following text is the content of a one-hour meeting. Please write a summary of this meeting in 500 characters or less."

[0084] Step 5:

[0085] The server provides the generated summary to the user.

[0086] The server receives the summary returned by the generative AI model and provides it to the user. The user can review the summary through the system's interface. The input is the summary returned by the generative AI model, and the output is the summary provided to the user. The server associates the summary with the user's account and stores it for later access by the user.

[0087] (Application example 1)

[0088] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Conventional meeting information management systems simply store recorded meeting data, making it difficult to quickly extract necessary information later. Furthermore, they lack a function to summarize and report the contents of meetings, which means that users have to spend a lot of time reviewing the entire contents of a meeting. This can lead to a decrease in efficiency, especially in situations where quick decision-making is required, such as in factories.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0091] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input, extracting information related to specific content, presenting the extracted information to the user, summarizing the extracted information, and reporting the summarized information to the user. This makes it possible to quickly extract necessary information from recorded conference data and summarize and report it.

[0092] "Multiple conference information" refers to video and audio data recorded in multiple conferences.

[0093] "Means of receiving input" refers to the interfaces and processes used to input meeting information into the system.

[0094] "Means for extracting information related to specific content" refers to algorithms or techniques for identifying and extracting information related to specific topics or keywords from meeting information.

[0095] "Means for presenting extracted information to a user" refers to an interface or method for providing extracted information to a user visually or audibly.

[0096] "Means for summarizing extracted information" refers to algorithms or techniques for concisely summarizing extracted information.

[0097] "Means for reporting summarized information to the user" refers to the interface or method for notifying the user of the summarized information.

[0098] A system for implementing this invention receives multiple pieces of conference information (video and audio recordings) as input, extracts information related to specific content, summarizes the extracted information, and reports it to the user. A specific embodiment of this system is described below.

[0099] Hardware and software used

[0100] Hardware:

[0101] Factory robots (e.g. KUKA, Fanuc, etc.)

[0102] Audio Capture Device

[0103] Computer (may be built into the robot)

[0104] software:

[0105] Python

[0106] moviepy (video processing library)

[0107] speech_recognition (speech recognition library)

[0108] transformers (AI models for generating Hugging Face)

[0109] Data processing and calculation

[0110] 1. Extract audio from video:

[0111] The server uses moviepy to extract audio from the conference video and generate a temporary audio file, which is used for subsequent speech recognition processing.

[0112] 2. Speech Recognition:

[0113] The server uses the speech_recognition library to convert the extracted speech into text, which is necessary to handle the contents of the meeting as text information.

[0114] 3. Text Summary:

[0115] The server uses Hugging Face's transformers library to summarize the text using a generative AI model (BART), and this summarized text is reported back to the user.

[0116] 4. Information Presentation and Reporting:

[0117] The server provides an interface for presenting the summarized information to the user, specifically by sending a notification to the user's terminal and displaying the summarized meeting content.

[0118] Specific examples

[0119] Factory managers input the video data of weekly production meetings into a robot. The robot analyzes the video data, summarizes important topics and decisions, and reports them to managers. This saves managers the time of having to review the entire meeting content, allowing them to quickly grasp only the important information.

[0120] Prompt Sentence Examples

[0121] An example of a prompt to input to a generative AI model is as follows:

[0122] "Please summarize the following text:"

[0123] Text extracted from meeting recordings

[0124] This prompt is used to ask the generative AI model to create a summary.

[0125] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0126] Step 1:

[0127] The server receives a conference video file from the user. The input is the conference video file, and the output is the saving of the video file. Specifically, the user uploads the video file recorded using the conference tool to the server.

[0128] Step 2:

[0129] The server uses moviepy to extract audio from conference videos. The input is the conference video file, and the output is an audio file. Specifically, the server reads the video file, separates the audio track, and generates a temporary audio file.

[0130] Step 3:

[0131] The server uses the speech_recognition library to convert the extracted audio into text. The input is an audio file, and the output is text data. Specifically, the server passes the audio file to a speech recognition engine, which converts the audio data into text information.

[0132] Step 4:

[0133] The server uses Hugging Face's transformers library to summarize text using a generative AI model (BART). The input is text data, and the output is summarized text. Specifically, the server inputs text data into the generative AI model and obtains the summarized text.

[0134] Step 5:

[0135] The server provides an interface for reporting the summarized information to the user. The input is the summary text, and the output is a notification to the user terminal. In concrete terms, the server sends the summary text to the user terminal and displays the notification.

[0136] Example 2

[0137] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] Conventional meeting information management systems make it difficult for users to quickly and accurately extract information related to specific content. Furthermore, the process of converting audio data into text data, analyzing that text data, and extracting relevant information is complicated, making it difficult to provide appropriate information in response to user inquiries. This makes it difficult to efficiently grasp the content of meetings, hindering work efficiency.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0140] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for converting the audio data into text data, means for analyzing the text data and extracting related information based on a user's inquiry, and means for presenting the extracted information to the user, thereby enabling the user to quickly and accurately extract information related to specific content.

[0141] "Meeting information" refers to video and audio data that records the contents of the meeting.

[0142] "Content-specific information" refers to information related to a particular topic or question that interests a user.

[0143] "Audio data" refers to digital audio files that record statements and discussions made during a meeting.

[0144] "Text data" refers to data obtained by converting voice data into character information.

[0145] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into corresponding text data.

[0146] "Natural language processing technology" refers to the technology of analyzing text data, understanding meaning and context, and extracting information.

[0147] "User Query" means a question or request entered by a User into the System requesting specific information.

[0148] "Means for extracting related information" refers to methods and technologies for searching and extracting related information from text data based on a user's inquiry.

[0149] "Means for presenting information" refers to the methods and techniques used to display extracted information to the user.

[0150] MODE FOR CARRYING OUT THE INVENTION

[0151] This invention relates to a system that receives meeting information (video and audio recordings) as input from a user, extracts information related to specific content, and presents it to the user. A specific embodiment of this system is described below.

[0152] System configuration

[0153] Hardware and software configuration:

[0154] 1. Server:

[0155] The server receives the voice data and converts it into text data using voice recognition technology.

[0156] The server analyzes the converted text data using natural language processing technology and extracts relevant information based on the user's inquiry.

[0157] The server presents the extracted information to the user.

[0158] 2. Terminal:

[0159] A user uses a terminal to upload the conference information to the server.

[0160] The user checks the information provided by the server through the terminal.

[0161] Specific techniques used:

[0162] As speech recognition technology, we use a general speech recognition API (e.g., Google Cloud Speech-to-Text API).

[0163] As a natural language processing technique, we use generative AI models (e.g., OpenAI's GPT-4 model).

[0164] System Operation

[0165] Upload audio data:

[0166] Users upload the audio data of the conference from their devices to the server. The audio data is typically provided in MP3 or WAV format. Users select the file through the system interface and click the upload button.

[0167] Audio data conversion:

[0168] The server receives the uploaded voice data and converts it into text data using voice recognition technology, extracting the spoken content from the voice data as text.

[0169] Analyzing text data:

[0170] The server then analyzes the converted text data using natural language processing technology, specifically using a generative AI model to extract information relevant to the user's inquiry.

[0171] Extracting relevant information:

[0172] The server searches for and extracts relevant comments from text data based on the user's inquiry. For example, in response to a query such as "What did that person say?", it extracts comments from a specific person.

[0173] Providing results:

[0174] The server returns the extracted relevant information to the user, who can check the results through their terminal.

[0175] Specific examples

[0176] If a user uploads audio from a meeting and asks, "What did they say about the progress of the project?", the system will proceed as follows:

[0177] 1. The user opens the browser on the terminal and accesses the system's web page. The user clicks the "Choose File" button and selects an audio file (e.g., "meeting_audio.mp3") from the local disk. Then, the user clicks the "Upload" button.

[0178] 2. When the server receives the voice data, it sends it to the speech recognition API. The API analyzes the voice data and returns the corresponding text data (e.g., "The project is 50% complete."). The server stores this text data.

[0179] 3. The server receives the user's query (e.g., "What did he say about the progress of the project?"). The server inputs the text data and the query into the generative AI model and extracts relevant utterances. For example, the server obtains a result such as, "Regarding the progress of the project, he said: 'Currently, progress is at 50%.'"

[0180] 4. The server sends the extracted information to the user's device. The user can check the results through the system interface. For example, a message like "The project progress was reported as follows: 'Currently, progress is 50%.'" will be displayed on the browser.

[0181] Example prompt sentence:

[0182] "I've uploaded the audio of the meeting. Can you tell me what they said about the project's progress?"

[0183] In this way, the server realizes a system that provides appropriate information in response to user inquiries.

[0184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0185] Program processing flow

[0186] Step 1: Upload your audio data

[0187] Specific behavior:

[0188] The user opens the browser on the terminal and accesses the system's web page. The user clicks the "Choose File" button and selects an audio file (e.g., "meeting_audio.mp3") from the local disk. Then, the user clicks the "Upload" button.

[0189] input:

[0190] An audio file selected by the user (e.g., "meeting_audio.mp3")

[0191] output:

[0192] Audio files uploaded to the server

[0193] Step 2: Receiving and storing audio data

[0194] Specific behavior:

[0195] The server receives the voice data uploaded by the user and stores it in a specified directory.

[0196] input:

[0197] User-uploaded audio files

[0198] output:

[0199] Audio files stored on the server

[0200] Step 3: Convert audio data to text

[0201] Specific behavior:

[0202] The server sends the stored voice data to a voice recognition API (e.g., a general voice recognition API), which analyzes the voice data and returns the corresponding text data.

[0203] input:

[0204] Audio files stored on the server

[0205] Data processing:

[0206] Convert voice data into text using speech recognition API

[0207] output:

[0208] Text data (e.g., "Project progress is 50%.")

[0209] Step 4: Save the text data

[0210] Specific behavior:

[0211] The server receives the text data returned from the speech recognition API and stores it in a database or file system.

[0212] input:

[0213] Text data returned from the speech recognition API

[0214] output:

[0215] Text data stored on the server

[0216] Step 5: Receiving user inquiries

[0217] Specific behavior:

[0218] A user enters a query through the system interface, for example, "What did they say about the progress of the project?"

[0219] input:

[0220] What the user is asking (e.g., "What did they say about the project status?")

[0221] output:

[0222] User inquiry sent to the server

[0223] Step 6: Analyze text data and extract relevant information

[0224] Specific behavior:

[0225] The server inputs the stored text data and the user's query into a generative AI model (e.g., OpenAI's GPT-4 model) to extract relevant information.

[0226] input:

[0227] Text data stored on the server, user inquiries

[0228] Data Calculation:

[0229] Analyze text data using generative AI models to extract relevant information

[0230] output:

[0231] Extracted relevant information (e.g., "Regarding the project progress, they said: 'Currently, it's 50% complete.'")

[0232] Step 7: Present the extracted information

[0233] Specific behavior:

[0234] The server sends the extracted relevant information to the user's terminal, and the user can check the results through the system interface.

[0235] input:

[0236] Extracted related information

[0237] output:

[0238] Relevant information displayed on the user's device (e.g., "Regarding the project's progress, they said: 'Currently, it's 50% complete.'")

[0239] In this way, the server realizes a system that provides appropriate information in response to user inquiries.

[0240] (Application example 2)

[0241] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] With conventional meeting information management systems, it was difficult to quickly and accurately extract specific content when searching for meeting details later. Furthermore, few systems utilized voice recognition or natural language processing technology, making it difficult to provide appropriate answers to user inquiries. Furthermore, they lacked the functionality to extract information across multiple meeting information sources, resulting in low user convenience.

[0243] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to the user, means for converting speech to text using speech recognition technology, and means for generating answers to user questions using natural language processing technology. This allows the user to quickly and accurately search for specific content and extract information across multiple pieces of conference information, improving convenience.

[0244] "Meeting information" refers to video and audio data recorded during a meeting.

[0245] "Information related to specific content" is a part of the conference information related to the content inquired by the user.

[0246] The "means for extracting information" refers to a technique or method for extracting information related to specific content from the conference information.

[0247] "Means for presenting to the user" refers to the techniques or methods for displaying the extracted information to the user.

[0248] "Speech recognition technology" is a technology that converts voice data into text data.

[0249] "Natural language processing technology" is a technology that analyzes text data and generates appropriate answers to user questions.

[0250] A "user question" is the content of an inquiry that a user makes to the system.

[0251] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[0252] The term "means for extracting information across multiple pieces of conference information" refers to a technique or method for extracting information related to specific content from multiple pieces of conference information.

[0253] A system for implementing this invention is configured as follows: The server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to a user, means for converting speech into text using speech recognition technology, and means for generating answers to user questions using natural language processing technology.

[0254] First, the server uses speech recognition technology to convert the audio data of the conference information into text data. For example, the "speech_recognition" library can be used as the speech recognition technology. The server reads the audio file and converts the audio data into text.

[0255] The server then uses natural language processing (NLP) techniques, such as the "pipeline" technique from the "transformers" library, to generate an answer to the user's question. Specifically, the server uses the "deepset / roberta-base-squad2" model to generate an appropriate answer to the user's question.

[0256] When a user wants to search for a specific topic, the server receives the user's question and uses speech recognition and natural language processing technologies to extract relevant information from the meeting information and present it to the user, allowing the user to search for the specific topic quickly and accurately.

[0257] As a concrete example, consider the case where a search is made for "discussions made about the installation of a new production line" during a meeting in a factory. When a user asks, "What was said about the installation of a new production line?", the server extracts relevant content from the meeting information and replies, "The installation of the new production line is scheduled to begin next month, with a budget of 5 million yen."

[0258] Example prompt sentence:

[0259] User: What did they say about setting up a new production line?

[0260] System: Installation of the new production line is scheduled to begin next month, with a budget of 5 million yen.

[0261] In this way, the server can combine speech recognition and natural language processing technologies to generate and present appropriate answers to user questions, enabling users to search for information across multiple conference information sources, improving convenience.

[0262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0263] Step 1:

[0264] The server receives multiple meeting information (video and audio recordings) as input. Specifically, the server receives audio and video files uploaded by users. This input data is used in subsequent processing steps.

[0265] Step 2:

[0266] The server uses speech recognition technology to convert the received voice data into text data. Specifically, it uses the "speech_recognition" library to read the audio file and convert the voice data into text. The input of this step is voice data, and the output is text data.

[0267] Step 3:

[0268] The server stores the converted text data, which is used in a subsequent natural language processing step, whose input is the text data and whose output is the stored text data.

[0269] Step 4:

[0270] When a user wants to search for something specific, they enter a question. Specifically, the user asks the system, "What did they say about installing a new production line?" The input for this step is the user's question, and the output is the question text.

[0271] Step 5:

[0272] The server uses natural language processing techniques to generate answers to user questions. Specifically, it uses the "pipeline" from the "transformers" library and the "deepset / roberta-base-squad2" model to generate appropriate answers to questions. The input for this step is the question text and stored text data, and the output is the generated answer.

[0273] Step 6:

[0274] The server presents the generated answer to the user. Specifically, it displays the answer on the user's screen. The input of this step is the generated answer, and the output is the answer presented to the user.

[0275] Step 7:

[0276] The user can review the presented answers and ask further questions if necessary. The input of this step is the answers presented to the user, and the output is the user's review.

[0277] In this way, the server can combine speech recognition and natural language processing technologies to generate and present appropriate answers to user questions, enabling users to search for information across multiple conference information sources, improving convenience.

[0278] Example 3

[0279] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] Conventional meeting information management systems make it difficult for users to quickly and accurately obtain specific information. They also lack the functionality to extract related information across multiple meeting records, preventing users from obtaining all the information they need at once. Furthermore, they lacked the means to present the extracted information in an easy-to-understand text format, resulting in low user convenience.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0282] In this invention, the server includes means for receiving multiple pieces of conference information as input and extracting information related to specific content, means for extracting information based on user inquiries, means for analyzing the extracted information using a natural language processing engine, means for converting the analyzed information into text format using a generative AI model, and means for presenting the converted text to the user. This allows the user to quickly and accurately obtain specific information and to simultaneously obtain related information across multiple pieces of conference information. Furthermore, presenting the extracted information in an easy-to-understand text format improves user convenience.

[0283] "Meeting information" refers to recorded data such as audio, video, and text related to the meeting.

[0284] "Means for extracting information" refers to techniques and methods for extracting information related to specific content from the meeting information.

[0285] "User Enquiry" refers to keywords or phrases that a user enters into the system to obtain specific information.

[0286] A "natural language processing engine" refers to software that analyzes user inquiries and extracted information and understands their meaning.

[0287] "Generative AI model" refers to an artificial intelligence model that converts analyzed information into natural language text format.

[0288] "Means for converting to text format" refers to the techniques and methods for converting extracted information into a text format that is easy for users to understand.

[0289] "Means for presenting to the user" refers to the technology or method for displaying the converted text on the user's device.

[0290] "Means for extracting information across multiple conference records" refers to techniques and methods for extracting related information from multiple conference records at once.

[0291] "Server" means a computer system that receives user queries and extracts, analyzes, converts, and presents information.

[0292] "Terminal" refers to a device through which a user can enter queries and receive results.

[0293] The present invention relates to a system for extracting conference information based on an inquiry from a user and presenting it in text format. A specific embodiment of this system will be described below.

[0294] First, this system uses the following hardware and software:

[0295] Hardware: Servers, terminals

[0296] Software: Database Management System (DBMS), Natural Language Processing (NLP) engine, Generative AI model

[0297] The server receives inquiries from users. When the user inputs specific keywords or phrases, the server sends the information to an NLP engine. The NLP engine extracts relevant meeting information from a database based on the input keywords or phrases. The extracted information is converted into text using a generative AI model. The server then sends the generated text to the user's device.

[0298] As a concrete example, the following processing occurs when a user asks, "Who are the participants in the next meeting?"

[0299] 1. The user types into the device interface, "Who's in my next meeting?"

[0300] 2. The device sends this query to the server.

[0301] 3. The server receives the query and sends it to the NLP engine.

[0302] 4. The NLP engine analyzes the keywords "next meeting" and "participants" and extracts relevant meeting information from the database.

[0303] 5. The server sends the extracted information to the generative AI model.

[0304] 6. The generative AI model generates the text, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[0305] 7. The server sends the generated text to the terminal.

[0306] 8. The device displays to the user, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[0307] In this way, the user can easily obtain the information he or she needs.

[0308] An example of a prompt sentence might be:

[0309] Who will be at your next meeting?

[0310] After entering this prompt, the system will present the user with a list of participants for the next meeting in text format. For example, the result might look like this: "The participants for the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[0311] This system allows users to quickly and accurately obtain specific information, and to simultaneously obtain related information across multiple conference information. Furthermore, by presenting the extracted information in an easy-to-understand text format, user convenience is improved. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0312] Step 1:

[0313] The user enters a query.

[0314] The user uses the device's interface to enter a query containing specific keywords or phrases, for example, "Who are the participants in the next meeting?" The entered data is temporarily stored in the device's memory.

[0315] Step 2:

[0316] The terminal sends a query to the server.

[0317] The terminal sends the query entered by the user to the server using a communication protocol such as an HTTP request. The input data is transferred to the server via the network.

[0318] Step 3:

[0319] The server receives the query and sends it to the NLP engine.

[0320] The server analyzes queries received from the device and sends them to the NLP engine. The server analyzes the received data and converts it into a format that the NLP engine can understand. The converted data is then passed to the NLP engine.

[0321] Step 4:

[0322] The NLP engine analyzes keywords and extracts information from the database.

[0323] The NLP engine analyzes the data received from the server and extracts specific keywords and phrases. For example, it searches the database based on keywords like "next meeting" or "participants." It then extracts relevant meeting information from the database and returns that information to the server.

[0324] Step 5:

[0325] The server sends the extracted information to the generative AI model.

[0326] The server sends the extracted information received from the NLP engine to the generative AI model. The server converts the extracted data into a format that the generative AI model can understand and passes it to the generative AI model.

[0327] Step 6:

[0328] The generative AI model converts it into text format.

[0329] The generative AI model converts the information received from the server into a text format that is easy for the user to understand. For example, it generates text such as, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato." The generated text is then returned to the server.

[0330] Step 7:

[0331] The server sends the generated text to the terminal.

[0332] The server sends the text received from the generative AI model to the device, again using a communication protocol such as an HTTP request. The generated text is then transferred to the device via the network.

[0333] Step 8:

[0334] The terminal displays the results to the user.

[0335] The terminal displays the text received from the server to the user. The user can check the results on the terminal screen. For example, the result may read, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[0336] (Application example 3)

[0337] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] There is a need to efficiently manage information on multiple meetings and discussions within a factory and quickly extract and present information related to specific keywords. However, conventional systems have had issues with cross-sectional searches of multiple meeting information, making it difficult to quickly obtain the necessary information. Furthermore, they lacked the functionality to present the extracted information along with the date, time, and participant information of related meetings, making it difficult for users to comprehensively grasp the information they need.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0340] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to a user, means for presenting the extracted information along with the date and time of related conferences and attendee information, means for extracting information across multiple pieces of conference information, means for a user to input specific keywords or phrases, and means for searching for information based on the keywords or phrases input by the user. This allows a user to efficiently manage information on multiple conferences and meetings within a factory and quickly obtain information related to specific keywords.

[0341] "Meeting information" refers to video and audio data recorded during meetings and discussions.

[0342] The "means for extracting information" refers to a method or device for extracting information related to specific content from input conference information.

[0343] "Means for presenting to the user" refers to a method or device for visually or audibly presenting the extracted information to the user.

[0344] The "means for presenting the date and time and participant information" refers to a method or device for providing the user with the date and time and participant information of the related conference together with the extracted information.

[0345] The "means for extracting information across multiple conferences" refers to a method or device for extracting related information collectively from multiple conference information.

[0346] A "means for entering keywords or phrases" is a method or device that allows a user to enter specific keywords or phrases into the system.

[0347] "Means for searching information" refers to a method or device for retrieving relevant information from conference information based on keywords or phrases entered by the user.

[0348] The system for implementing this invention aims to efficiently manage information on multiple meetings and conferences within a factory and quickly extract and present information related to specific keywords. A specific embodiment of this system will be described below.

[0349] System configuration

[0350] The system consists of the following major components:

[0351] 1. Database server: Manages the database for storing conference information (video and audio recordings).

[0352] 2. Search server: Searches conference information and extracts relevant information based on keywords and phrases provided by the user.

[0353] 3. User Interface: Provides an interface for users to enter keywords or phrases and view search results.

[0354] Hardware and software used

[0355] Hardware: Factory robots, servers

[0356] Software: SQLite (database management system), Python (programming language)

[0357] Data processing and calculation flow

[0358] 1. Database connection: The database server uses SQLite to store conference information, including video and audio recordings, conference dates and times, and participant information.

[0359] 2. Information extraction: The search server extracts relevant information from the conference information based on the keywords or phrases entered by the user, specifically by using SQL queries to search for detailed conference information.

[0360] 3. Result presentation: The extracted information is presented to the user through a user interface, including the meeting date, time, participants, and details.

[0361] Specific examples

[0362] For example, if a factory manager wants to search for past meeting information using the keyword "quality control," he or she would perform the following operations.

[0363] 1. Enter the keyword "quality control" into the user interface.

[0364] 2. The search server connects to the database server and searches for relevant conference information.

[0365] 3. The extracted information, along with the meeting date, time, and participant information, is displayed in the user interface.

[0366] Prompt Sentence Examples

[0367] "Find information about quality control discussed at past meetings."

[0368] By inputting this prompt into a generative AI model, relevant meeting information can be quickly retrieved.

[0369] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0370] Step 1:

[0371] A user uses a terminal to enter a particular keyword or phrase into a user interface.

[0372] Input: Keyword or phrase (e.g., "quality control")

[0373] Output: The keyword or phrase you entered

[0374] Step 2:

[0375] The terminal transmits the entered keywords or phrases to a search server.

[0376] Input: Keywords or phrases entered by the user

[0377] Output: Keywords or phrases sent to the search server

[0378] Step 3:

[0379] The search server connects to a database server and searches for relevant conference information based on the received keywords or phrases.

[0380] Input: Submitted keyword or phrase

[0381] Output: Search results using SQL queries (related conference information)

[0382] Step 4:

[0383] The database server searches the conference information and extracts relevant information based on the SQL query from the search server.

[0384] Input: SQL query

[0385] Output: Extracted meeting information (meeting date, time, participants, details)

[0386] Step 5:

[0387] The search server processes the extracted information received from the database server and converts it into a format for presentation to the user.

[0388] Input: Extracted meeting information

[0389] Output: Information transformed into a format for presentation to the user

[0390] Step 6:

[0391] The search server transmits the converted information to the terminal.

[0392] Input: Formatted information

[0393] Output: Information sent to the terminal

[0394] Step 7:

[0395] The terminal displays the received information on a user interface and presents it to the user.

[0396] Input: Received information

[0397] Output: Information displayed in the user interface (meeting date, time, participants, details)

[0398] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0399] "Example 1"

[0400] In one embodiment of the present invention, a recording of a meeting held by a user using a conferencing tool (e.g., Zoom) is input to the system. The system includes an emotion engine that recognizes emotions from the user's voice.

[0401] "Example 2"

[0402] When a user asks a question such as "What did that person say?", the system extracts relevant content from the input meeting information. This extraction is done using speech recognition and natural language processing technologies. Furthermore, the emotion engine recognizes emotions from the user's voice and extracts information based on those emotions.

[0403] "Example 3"

[0404] The system presents the extracted information to the user in text format, along with relevant meeting dates, participants, and other information. The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[0405] "Example 4"

[0406] When users query for specific keywords or phrases, the system extracts information related to those keywords or phrases from the meeting information.The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[0407] "Example 5"

[0408] When a user inquires about discussions that took place in multiple meetings, the system extracts relevant information from multiple meeting records. This allows users to obtain information across multiple meetings at once, rather than just one. The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[0409] The processing flow of each embodiment will be described below.

[0410] "Example 1"

[0411] Step 1: Users input recording data of meetings they have held using a conferencing tool (e.g., Zoom) into the system.

[0412] Step 2: The system activates an emotion engine that recognizes emotions from the user's voice.

[0413] Step 1: The user asks a question such as, "What did that person say?"

[0414] Step 2: The system extracts relevant content from the input meeting information using voice recognition and natural language processing technologies.

[0415] Step 3: The emotion engine recognizes emotions from the user's voice and extracts information based on those emotions.

[0416] "Example 3"

[0417] Step 1: The system presents the extracted information to the user in text format.

[0418] Step 2: Provide relevant meeting dates, times, participants, and other information.

[0419] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[0420] "Example 4"

[0421] Step 1: User queries for a specific keyword or phrase.

[0422] Step 2: The system extracts information related to the keyword or phrase from the meeting information.

[0423] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[0424] "Example 5"

[0425] Step 1: A user inquires about discussions that took place across multiple meetings.

[0426] Step 2: The system extracts relevant information from multiple conferences, allowing users to obtain information across multiple conferences at once, rather than just a single conference.

[0427] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[0428] Example 1

[0429] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0430] Conventional meeting recording systems simply store the contents of the meeting, making it difficult to grasp the emotional state of the participants during the meeting. This makes it difficult to properly evaluate the effectiveness of the meeting or the reactions of the participants, and makes it difficult to obtain specific feedback to improve the quality of the meeting.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0432] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input, separating the video data from the audio data, means for analyzing the audio data and recognizing emotions, means for storing the recognized emotions in a database, and means for presenting the stored emotion analysis results to the user, thereby enabling automatic analysis and visualization of the emotional states of participants during a conference.

[0433] "Meeting information" refers to video and audio data recorded during a meeting.

[0434] "Video data" refers to data that includes video information from a conference.

[0435] "Audio data" refers to data containing audio information during a conference.

[0436] "Means for separating" refers to the technical means for extracting video data and audio data separately.

[0437] "Means for analyzing" refers to the technical means for analyzing the audio data and extracting specific information.

[0438] "Means for recognizing emotions" refers to technical means for identifying emotional states from audio data.

[0439] "Database" refers to a data storage system for storing analytical results and other information.

[0440] "Presentation Means" refers to the technical means by which information is displayed or presented to a user.

[0441] "User" refers to any individual or organization that uses the System.

[0442] "Query" refers to a request for information made by a user to a system.

[0443] "Means for extracting information across multiple conferences" refers to technical means for comparing and analyzing multiple conference information to extract common or related information.

[0444] MODE FOR CARRYING OUT THE INVENTION

[0445] This invention is a system that receives multiple pieces of conference information (video and audio recordings) as input, separates the video data from the audio data, analyzes the audio data to recognize emotions, and presents the results to the user.

[0446] Hardware and software used

[0447] The server uses the following hardware and software:

[0448] Hardware: powerful processor, sufficient memory, storage devices, and network interfaces

[0449] Software: FFmpeg (video processing library), Google Cloud Speech-to-Text API, IBM Watson (registered trademark) sentiment analysis API, database management system (e.g., MySQL (registered trademark))

[0450] Data processing and calculation

[0451] The server receives the meeting recording data uploaded by the user and performs the following processes:

[0452] 1. Separating video and audio data: Use FFmpeg to separate video and audio data. The audio data is converted to WAV format.

[0453] 2. Analyze voice data: Use Google Cloud Speech-to-Text API and IBM Watson sentiment analysis API to analyze voice data and identify the user's emotional state.

[0454] 3. Save to database: The analysis results are saved to a database, including the meeting timestamp, emotion type, and emotion intensity.

[0455] 4. Presentation to the user: Based on the saved sentiment analysis results, feedback is provided to the user. The feedback is generated as a report visualizing changes in sentiment and displayed on the user's dashboard.

[0456] Specific examples

[0457] As a concrete example, consider a scenario in which a user uploads recording data of a meeting held using a conferencing tool to the system. When the user uploads the recording data of the meeting to the system, the server receives the data and inputs the voice data into the emotion engine. The emotion engine recognizes the user's emotion from the voice data and stores the result in a database. Finally, the user can receive a report visualizing the changes in emotion during the meeting.

[0458] Prompt Sentence Examples

[0459] Users upload recordings of meetings held using a conferencing tool to the system. The system analyzes the audio data and uses an emotion engine to recognize the user's emotions. The analysis results are stored in a database and a report is generated that visualizes changes in emotions.

[0460] In this way, it is possible to specifically explain how the server, terminal, and user work together to utilize the system.

[0461] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0462] Step 1:

[0463] A user uploads a meeting recording.

[0464] The user accesses the system's web interface, selects the conference recording data, and clicks the upload button. Once the upload is complete, the server receives the data.

[0465] Input: Meeting recording data (video and audio recording)

[0466] Output: Meeting recording data saved on the server

[0467] Step 2:

[0468] The server receives the recorded data and separates the video data from the audio data.

[0469] The server temporarily stores the received recording data and calls the FFmpeg library to separate the video data from the audio data. The audio data is saved as a WAV file.

[0470] Input: Meeting recording data stored on the server

[0471] Output: Separated video and audio data (WAV format)

[0472] Step 3:

[0473] The server inputs the voice data into the emotion engine.

[0474] The server sends the stored voice data to the emotion engine, which receives the voice data and starts analyzing it.

[0475] Input: Separated audio data (WAV format)

[0476] Output: Voice data input to the emotion engine

[0477] Step 4:

[0478] The emotion engine analyzes voice data and recognizes emotions.

[0479] The emotion engine analyzes the audio data and identifies the user's emotion at each timestamp, for example, recognizing that the user is feeling "joyful" five minutes into the meeting.

[0480] Input: Speech data input to the emotion engine

[0481] Output: Parsed emotion data (timestamp, emotion type, emotion intensity)

[0482] Step 5:

[0483] The server stores the analysis results in a database.

[0484] The server stores the analysis results received from the emotion engine in a database, including the meeting timestamp, emotion type, and emotion intensity.

[0485] Input: Parsed emotion data

[0486] Output: Sentiment analysis results stored in a database

[0487] Step 6:

[0488] The server provides feedback on the emotion analysis results to the user.

[0489] The server provides feedback to the user based on the emotion analysis results stored in the database. The feedback is generated as a report visualizing changes in emotion and displayed on the user's dashboard.

[0490] Input: Sentiment analysis results stored in the database

[0491] Output: Sentiment analysis results presented to the user (visual report)

[0492] (Application example 1)

[0493] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0494] With conventional conferencing systems, it was difficult to grasp the emotions of participants in real time during a meeting, and it was particularly difficult to detect abnormal emotional changes (e.g., anger, fear, stress) and take appropriate measures. Furthermore, it was not possible to take user emotions into consideration when extracting information related to specific content from recorded meeting data, making it impossible to improve the quality and safety of meetings.

[0495] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0496] In this invention, the server includes a means for receiving multiple pieces of conference information (video and audio recordings) as input, extracting information related to specific content, presenting the extracted information to the user, recognizing emotions from the user's voice, and issuing an alert if the recognized emotion is abnormal. This makes it possible to grasp the emotions of participants during a conference in real time, detect abnormal emotional changes, and take appropriate action. Furthermore, the user's emotions can be taken into consideration when extracting information related to specific content from the conference recording data, thereby improving the quality and safety of the conference.

[0497] "Meeting information" refers to video and audio data recorded during a meeting.

[0498] "Information related to a specific content" refers to information extracted from the conference information that is related to a specific topic or theme that is of interest to the user.

[0499] The term "extraction means" refers to a technical means for extracting information related to specific content from the conference information.

[0500] "Presentation means" refers to the technical means by which the extracted information is presented to the user visually or audibly.

[0501] "Means for recognizing emotions" refers to technical means for analyzing emotions from a user's voice data and identifying a specific emotional state.

[0502] "Means for issuing an alert" refers to a technical means for issuing a warning to a user or administrator when a recognized emotion is abnormal.

[0503] The system for implementing this invention receives multiple meeting information (video and audio recordings) as input, extracts information related to specific content, and presents it to the user. It also has the function of recognizing emotions from the user's voice and issuing an alert if the recognized emotion is abnormal.

[0504] Hardware and software used

[0505] Hardware:

[0506] server

[0507] Smartphone or smart glasses

[0508] microphone

[0509] software:

[0510] pyaudio library (audio recording)

[0511] wave library (storing audio data)

[0512] emotion_recognition library (emotion recognition)

[0513] System processing overview

[0514] Audio recording

[0515] The device (smartphone or smart glasses) records the audio during the meeting. It uses the pyaudio library to capture audio data in real time and save it in WAV format.

[0516] emotion recognition

[0517] The server receives the recorded audio data and analyzes the emotions using the emotion_recognition library, which analyzes the audio data and identifies the user's emotional state.

[0518] Information Extraction and Presentation

[0519] The server extracts information related to a specific content from the conference information, and the extracted information is presented to the user visually or audibly.

[0520] Alert occurs

[0521] The server issues an alert if the recognized emotion is abnormal (e.g., anger, fear, stress), which is then notified to the user or administrator.

[0522] Specific examples

[0523] For example, if a participant suddenly becomes angry during a meeting, the device will record the audio and send it to the server. The server will then analyze the recorded audio and recognize the emotion of "anger." If the recognized emotion is abnormal, the server will send an alert to the administrator and prompt them to take appropriate action.

[0524] Prompt Sentence Examples

[0525] Analyze the audio data recorded during the meeting and recognize the emotions of the participants. If emotions such as "anger," "fear," and "stress" are detected, an alert will be issued.

[0526] In this way, it is possible to grasp the emotions of participants in a meeting in real time, detect abnormal emotional changes, and take appropriate action.In addition, when extracting information related to specific content from recorded meeting data, user emotions can be taken into consideration, improving the quality and safety of meetings.

[0527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0528] Step 1:

[0529] The terminal records audio during a conference. It uses the pyaudio library to capture audio data in real time and save it in WAV format. The input is the audio signal from the microphone, and the output is a WAV audio file. Specifically, the terminal reads the audio data at regular intervals and saves it in a buffer.

[0530] Step 2:

[0531] The terminal sends the recorded audio data to the server. The input is a WAV format audio file, and the output is the transfer of audio data to the server. Specifically, the terminal uploads the audio file to the server via the network.

[0532] Step 3:

[0533] The server analyzes the received audio data and recognizes emotions. It uses the emotion_recognition library to analyze the audio data and identify the user's emotional state. The input is a WAV audio file, and the output is the recognized emotion (e.g., anger, fear, stress). Specifically, the server inputs the audio data into the library and runs the emotion recognition model.

[0534] Step 4:

[0535] The server determines whether the recognized emotion is abnormal. The input is the recognized emotion, and the output is the abnormal emotion determination result. Specifically, the server compares the recognized emotion with a predefined list of abnormal emotions, and if there is a match, it determines that the emotion is abnormal.

[0536] Step 5:

[0537] The server issues an alert when an abnormal emotion is detected. The input is the abnormal emotion judgment result, and the output is an alert notification. Specifically, the server sends an email or push notification to the administrator to inform them that an abnormal emotion has been detected.

[0538] Step 6:

[0539] The server extracts information related to specific content from the meeting information. The input is the meeting information (video and audio recordings), and the output is the extracted information. Specifically, the server uses natural language processing technology to extract information related to specific topics or themes that interest the user from the meeting information.

[0540] Step 7:

[0541] The server presents the extracted information to the user. The input is the extracted information, and the output is the presentation of the information to the user. In concrete terms, the server displays the extracted information through an interface for providing it to the user visually or audibly.

[0542] Example 2

[0543] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0544] Conventional meeting information management systems have made it difficult for users to quickly search for specific content, especially when extracting information from audio data. Furthermore, they do not provide information that takes into account the user's emotions, which means they are unable to provide information that meets the user's needs.

[0545] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a plurality of pieces of conference information as input and extracting information related to specific content, means for presenting the extracted information to the user, means for receiving a voice inquiry from the user and converting the voice into text, means for extracting related information based on the converted text, and means for recognizing emotions from the user's voice and filtering information based on the emotions. This enables the user to quickly and accurately obtain information related to specific content, and further enables information to be provided according to the user's emotions.

[0546] "Meeting information" refers to data such as audio, video, and text recorded during a meeting.

[0547] "Information related to specific content" refers to information about specific topics or comments within the meeting information that is extracted based on a user's inquiry.

[0548] The term "extraction means" refers to a technique or method for extracting information related to specific content from the conference information.

[0549] "Means of presentation to the user" refers to the technology or method used to display or audibly convey the extracted information to the user.

[0550] "Voice inquiry" refers to a question or request made by a user to a system using voice.

[0551] "Means for converting speech to text" refers to the technology or method for converting speech data into text data.

[0552] "Means for extracting relevant information" refers to techniques and methods for extracting information relevant to the user's inquiry from the converted text data.

[0553] "Means for recognizing emotions" refers to technologies and methods for analyzing emotions from a user's voice and identifying those emotions.

[0554] "Information filtering means" refers to techniques and methods for filtering extracted information in a way that is most relevant to the user based on perceived emotions.

[0555] MODE FOR CARRYING OUT THE INVENTION

[0556] The present invention provides a system that allows a user to quickly and accurately search for specific content from conference information and provides information that corresponds to the user's emotions. A specific embodiment of this system will be described below.

[0557] Hardware and software used

[0558] Hardware: Servers, devices (PCs, smartphones, etc.)

[0559] Software: speech recognition software (e.g., Google Speech-to-Text API), natural language processing software (e.g., OpenAI GPT-4), emotion recognition engines (e.g., IBM Watson Tone Analyzer)

[0560] System Operation

[0561] 1. User Inquiries

[0562] The user uses the device to make a voice inquiry such as, "What did that person say?" For example, the user speaks into the microphone of a smartphone.

[0563] 2. Sending audio data

[0564] The device sends the user's voice data to a server, and the voice data recorded by the smartphone is uploaded to the server via the Internet.

[0565] 3. Voice Recognition

[0566] The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the voice data into text data. The server receives the voice data and calls the Google Speech-to-Text API to convert it from voice to text.

[0567] 4. Natural Language Processing

[0568] The server inputs the converted text data into natural language processing software (e.g., OpenAI GPT-4) to extract relevant content from the meeting information.The server inputs the text data into OpenAI GPT-4 to extract relevant information for the question, "What did that person say?"

[0569] 5. Emotion recognition

[0570] The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the user's voice. The server inputs the voice data into IBM Watson Tone Analyzer and recognizes that the user's emotion is "confused."

[0571] 6. Information Extraction

[0572] The server then filters the information extracted by the natural language processing software based on the recognized emotion and selects the most relevant information for the user.The server selects the clearest and most specific answer based on the emotion "confused."

[0573] 7. Submitting the results

[0574] The server sends the selected information to the device and provides feedback to the user by displaying or speaking. The server sends the selected text to the smartphone, which displays the text on the screen or reads it aloud.

[0575] Specific examples

[0576] For example, if a user asks a question to a terminal during a meeting, such as "What did that person say?", the following specific actions are performed.

[0577] 1. The user speaks into the smartphone microphone, "What did that person say?"

[0578] 2. The smartphone uploads the recorded audio data to a server via the Internet.

[0579] 3. The server receives the voice data and calls the Google Speech-to-Text API to convert it from voice to text.

[0580] 4. The server inputs the text data into OpenAI GPT-4 and extracts relevant information for the question, "What did that person say?"

[0581] 5. The server inputs the voice data into IBM Watson Tone Analyzer and determines that the user's emotion is "confusion."

[0582] 6. The server selects the most clear and specific answer based on the emotion of "confused."

[0583] 7. The server sends the selected text to the smartphone, which displays it on the screen or reads it aloud.

[0584] Prompt Sentence Examples

[0585] An example of a prompt sentence to input to the generative AI model is as follows:

[0586] When a user asks, "What did that person say?", extract relevant utterances from the meeting text data. Provide a clear answer, especially if the user's emotion is "confused."

[0587] Using this prompt, the generative AI model can generate an appropriate answer to the user's question.

[0588] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0589] Step 1:

[0590] The user makes a voice inquiry to the device. Specifically, the user speaks into the smartphone microphone, saying, "What did that person say?" The input is the user's voice data, and the output is an audio file recorded on the device.

[0591] Step 2:

[0592] The device sends the recorded audio data to a server. Specifically, the audio data recorded by the smartphone is uploaded to the server via the Internet. The input is the audio file recorded on the device, and the output is the audio data sent to the server.

[0593] Step 3:

[0594] The server uses speech recognition software to convert the voice data into text data. Specifically, the server calls the Google Speech-to-Text API to perform "voice-to-text" conversion. The input is the voice data sent to the server, and the output is the converted text data.

[0595] Step 4:

[0596] The server inputs the converted text data into natural language processing software to extract relevant content from the meeting information. Specifically, the server uses OpenAI GPT-4 to extract relevant information for the question, "What did that person say?" The input is the converted text data, and the output is the extracted relevant information.

[0597] Step 5:

[0598] The server uses an emotion recognition engine to recognize emotions from the user's voice. Specifically, the server uses IBM Watson Tone Analyzer to recognize that the user's emotion is "confused." The input is the voice data sent to the server, and the output is the recognized emotion data.

[0599] Step 6:

[0600] The server filters the information extracted by the natural language processing software based on the recognized emotion and selects the most relevant information for the user. Specifically, the server selects the clearest and most specific answer based on the emotion "confused." The input is the extracted relevant information and the recognized emotion data, and the output is the selected information.

[0601] Step 7:

[0602] The server sends the selected information to the device and provides feedback to the user by display or voice. Specifically, the server sends the selected text to the smartphone, which then displays the text on the screen or reads it aloud. The input is the selected information, and the output is the information presented to the user.

[0603] (Application example 2)

[0604] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0605] In autonomous vehicles, there is a need to be able to review conversations that took place in the car later, but current systems do not adequately provide such functionality. Furthermore, there is a demand for more appropriate information provision by providing information based on the user's emotions. This calls for a system that can search for the content of conversations in the car and provide information based on emotions.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0607] In this invention, the server includes: means for accepting a user's inquiry; means for extracting information related to the inquiry from video data or audio data representing multiple conferences or conversations using a prompt sentence instructing the extraction of information related to the inquiry from video data or audio data representing multiple conferences or conversations and a generative AI model; and means for providing the extracted information related to the inquiry to the user. The server further includes means for recognizing a user's emotion using an emotion engine, and the means for providing to the user provides the extracted information related to the inquiry to the user using a prompt sentence instructing the extraction of information related to the inquiry to the user based on the recognized user's emotion and the generative AI model. The multiple conferences or conversations took place in a vehicle. This makes it possible to later check the conferences or conversations that took place in the vehicle, and to provide appropriate information based on the user's emotion.

[0608] "Meeting information" refers to the contents of a meeting recorded in the form of audio, video, or the like.

[0609] "Information related to specific content" refers to a specific part of the meeting information that is extracted based on a user's inquiry or search conditions.

[0610] "Voice recognition technology" refers to technology that converts voice data into text data.

[0611] "Texting" refers to the process of converting non-text data, such as audio or video, into written information.

[0612] "User inquiry" refers to a question or search request made by a user to the system.

[0613] An "emotion engine" is a technology that analyzes emotions from a user's voice or text and provides appropriate information based on those emotions.

[0614] "In-vehicle conversations" refers to voice communications conducted by drivers and passengers inside an autonomous vehicle.

[0615] "Search" refers to the process of finding relevant information from a database or recorded information based on specific criteria.

[0616] "Providing information" means presenting appropriate answers and related information to users' inquiries.

[0617] A system for implementing this invention has the following configuration. First, conversations taking place inside the vehicle are collected using a microphone installed inside the vehicle. The collected voice data is sent to an on-board computer and converted into text data using voice recognition technology. This voice recognition technology uses the SpeechRecognition library.

[0618] The converted text data is then analyzed using natural language processing (NLP) techniques, using the Transformers library and the deepset / roberta-base-squad2 model. When a user issues a query to the system, NLP techniques are used to search for and extract relevant information.

[0619] Furthermore, the system analyzes the user's emotions using an emotion engine. The TextBlob library is used for this emotion analysis. Based on the results of the emotion analysis, the system provides appropriate information to the user.

[0620] As a specific example, if a user asks, "We were talking about what to eat at the next rest stop. What did they say?" the system will search for relevant content from multiple meetings or conversations and provide relevant information.

[0621] An example of a prompt is as follows:

[0622] After inputting video or audio data representing multiple meetings or conversations, the system may ask, "User question: We were talking about what to eat at the next rest stop. What did they say?" Extract information relevant to the inquiry from video or audio data representing multiple meetings or conversations.

[0623] Output of the generative AI model: The conversation went something like, "Let's eat ramen at the next rest stop."

[0624] In this way, multiple conversations that took place in the car can be reviewed later. The server uses a generative AI model and prompts that instruct the server to provide information related to the extracted query to the user based on the recognized user's sentiment. Specifically, if the user's sentiment is positive, the system provides more detailed information, and if it is negative, it provides concise information. This makes it possible to provide appropriate information based on the user's sentiment.

[0625] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0626] Step 1:

[0627] A microphone installed inside the vehicle collects video or audio data representing multiple conversations taking place inside the vehicle.

[0628] Input: In-car video data or audio data

[0629] Output: Collected video or audio data

[0630] How it works: A microphone collects audio from inside the car in real time and transmits it to the onboard computer.

[0631] Step 2:

[0632] The onboard computer converts the collected voice data into text data using voice recognition technology.

[0633] Input: Collected audio data

[0634] Output: Text data

[0635] Specific operation: The vehicle's computer uses the SpeechRecognition library to convert voice data into text data.

[0636] Step 3:

[0637] The onboard computer analyzes the converted text data using natural language processing technology.

[0638] Input: Text data

[0639] Output: Analysis results

[0640] What it does: The onboard computer uses the Transformers library to parse the text data.

[0641] Step 4:

[0642] The user queries the system.

[0643] Input: User query

[0644] Output: Inquiry details

[0645] Specific action: The user asks the system a question via voice or text.

[0646] Step 5:

[0647] A server or on-board computer searches and extracts relevant information based on the user's query.

[0648] Input: Inquiry details, analysis results

[0649] Output: Search results

[0650] Specific behavior: The server or on-board computer uses a prompt statement that instructs the extraction of information related to the inquiry from the text data and a generative AI model to search for and extract information related to the inquiry content from the text data.

[0651] Step 6:

[0652] The server or on-board computer uses an emotion engine to analyze the user's emotions.

[0653] Input: User voice or text

[0654] Output: Emotion analysis results

[0655] Specific operation: The server or on-board computer uses the TextBlob library to analyze emotions from the user's voice or text.

[0656] Step 7:

[0657] The server or the in-vehicle computer provides appropriate information to the user based on the emotion analysis results.

[0658] Input: Search results, sentiment analysis results

[0659] Output: Information provided

[0660] Specific operation: The server or on-board computer uses prompts based on the results of emotion analysis and a generative AI model to provide the user with appropriate information in voice or text.

[0661] Example 3

[0662] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0663] Conventional meeting information management systems make it difficult for users to quickly and accurately obtain specific information. Furthermore, they do not provide information based on the user's emotions, which results in a poor user experience. Furthermore, they lack the functionality to extract related information across multiple meeting information, making it difficult for users to obtain the information they need all at once.

[0664] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0665] In this invention, the server includes means for receiving multiple pieces of conference information as input and extracting information related to specific content, means for presenting the extracted information to the user, means for extracting information based on a user inquiry, means for recognizing the user's emotions and extracting information based on the emotions, and means for presenting the extracted information to the user in text format. This allows the user to quickly and accurately obtain the information they need, and also makes it possible to provide information according to the user's emotions. Furthermore, since related information can be extracted across multiple pieces of conference information, the user can obtain the information they need all at once.

[0666] "Meeting information" refers to data such as audio, video, and text recorded during a meeting.

[0667] "Means for extracting information" refers to techniques and methods for extracting information related to specific content from the meeting information.

[0668] "Means of presentation to the user" refers to the techniques and methods for providing the extracted information to the user visually or audibly.

[0669] "User Inquiry" refers to a question or request made by a user to the system to obtain specific information.

[0670] "Means of emotion recognition" refers to technologies and methods for analyzing and identifying emotions from user input and behavior.

[0671] "Text format" refers to a format in which information is expressed as a string of characters.

[0672] "Means for extracting information across multiple sources" refers to techniques and methods for extracting related information from multiple conference information sources.

[0673] MODE FOR CARRYING OUT THE INVENTION

[0674] The system of the present invention receives multiple pieces of conference information as input, extracts information related to specific content, and presents it to the user. This system has the function of extracting information based on user inquiries and further recognizing the user's emotions and extracting information based on those emotions. Specific embodiments are described below.

[0675] Hardware and software used

[0676] 1. Server

[0677] The server is connected to a database (such as MySQL) that stores conference information.

[0678] The server runs a Python script to extract information from the database.

[0679] The server uses an emotion engine such as IBM Watson Tone Analyzer for emotion recognition.

[0680] 2. Terminal

[0681] Users access the system using devices such as PCs and smartphones.

[0682] The terminal provides a user interface and accepts inquiries from the user.

[0683] Data processing and calculation

[0684] 1. Information Extraction

[0685] The server receives user queries and executes Python scripts to send queries to a MySQL database.

[0686] The server extracts information related to specific keywords or phrases from a database.

[0687] Example: In response to the query "When is the next meeting?", the date and time of the next meeting is retrieved from the database.

[0688] 2. Emotional Recognition

[0689] The server sends the user's input text to the IBM Watson Tone Analyzer to analyze the user's emotions.

[0690] The server receives the analysis results and recognizes that the user's emotions are "interest" or "concern."

[0691] 3. Presentation of Information

[0692] The server formats the extracted information in text format and returns it to the user.

[0693] Example: Generate a message saying "The next meeting will be held on October 15, 2023 at 2:00 PM" and send it to the user's device.

[0694] Specific examples

[0695] When the user asks, "When is the next meeting?" the server responds, "The next meeting will be on October 15, 2023 at 2:00 p.m."

[0696] When the user asks, "What were the main discussions in the last three meetings?" the server responds, "The main discussions in the last three meetings were: 1. New product development 2. Marketing strategy 3. Budget review."

[0697] Prompt Sentence Examples

[0698] "When is the next meeting?"

[0699] "What were the main discussions at the last three meetings?"

[0700] This system is designed to provide information quickly and accurately in response to user inquiries. In addition, by using an emotion engine, it is possible to provide appropriate information according to the user's emotions. Furthermore, since it is possible to extract related information across multiple conference information, the user can obtain the information they need at once. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[0701] Step 1:

[0702] User inquiries

[0703] The user accesses the system interface using a device (PC or smartphone), enters a specific keyword or phrase in the text box, and clicks the send button.

[0704] Input: A keyword or phrase typed by the user (e.g., "When is our next meeting?")

[0705] Output: The user's query is sent to the server.

[0706] Step 2:

[0707] Information Extraction

[0708] The server receives user input and runs a Python script to query a MySQL database, extracting information related to specific keywords or phrases.

[0709] Input: User's inquiry

[0710] Data manipulation: A Python script queries the database and extracts relevant information (e.g., "SELECT date FROM meetings WHERE id = (SELECT MAX(id) FROM meetings)").

[0711] Output: Extracted information (e.g., next meeting date and time)

[0712] Step 3:

[0713] Emotion Engine Operation

[0714] The server sends the user's input text to the emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotion. The server receives the analysis results and recognizes the user's emotion.

[0715] Input: User-entered text

[0716] Data calculations: The sentiment engine analyzes the text and identifies sentiment (e.g., "interest" or "concern").

[0717] Output: User sentiment analysis results

[0718] Step 4:

[0719] Presentation of information

[0720] The server formats the extracted information in text format and responds to the user, providing appropriate information according to the user's emotions.

[0721] Input: Extracted information and sentiment analysis results

[0722] Data processing: Format the extracted information into text and generate an appropriate message for the user (e.g., "The next meeting will be held on October 15, 2023 at 2:00 PM").

[0723] Output: Messages sent to the user's terminal

[0724] Step 5:

[0725] User Visibility

[0726] The user's device receives the message sent from the server and displays it on the screen, allowing the user to check the necessary information.

[0727] Input: Message sent from the server

[0728] Output: Information displayed on the user's device (e.g., "The next meeting will be held on October 15, 2023 at 2:00 PM.")

[0729] In this way, users can quickly and accurately obtain the information they need. In addition, by using an emotion engine, it is possible to provide appropriate information according to the user's emotions.

[0730] (Application example 3)

[0731] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0732] Conventional meeting information management systems made it difficult for users to quickly obtain specific information, and were unable to provide integrated information across multiple meetings. Furthermore, they did not take user emotions into consideration, making it difficult to provide information tailored to user needs. Furthermore, factory robots could not optimize their work based on past meeting information, making it difficult to achieve efficient work.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0734] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to the user, means for recognizing the user's emotions and extracting information based on those emotions, means for presenting the extracted information in text format, and means installed in the factory robot for optimizing work based on past conference information. This allows users to quickly obtain the information they need, enables information to be provided based on emotions, and enables factory robots to work more efficiently.

[0735] "Meeting information" refers to video and audio data that records the contents of the meeting.

[0736] The term "means for extracting information" refers to a method or device for extracting information related to a specific content from the conference information.

[0737] "Means for presenting to the user" refers to a method or device for showing the extracted information to the user.

[0738] "Means for recognizing emotions" refers to a method or device for analyzing a user's emotions and understanding those emotions.

[0739] "Means for presenting in text form" refers to a method or device for displaying extracted information to a user as text information.

[0740] A "factory robot" refers to an automated mechanical device designed to perform tasks in a factory.

[0741] "Means for optimizing work" refers to methods and devices that enable factory robots to perform work efficiently based on past meeting information.

[0742] In order to implement the present invention, the following hardware and software are used.

[0743] Hardware

[0744] Factory Robots

[0745] Audio input device

[0746] display

[0747] software

[0748] EmotionEngine (emotion recognition engine)

[0749] MeetingDataExtractor (meeting information extraction engine)

[0750] System configuration

[0751] The server receives multiple meeting information (video and audio recordings) as input and has a means to extract information related to specific content. When a user queries for a specific keyword or phrase, the server uses the EmotionEngine to recognize the user's emotion and extracts information based on that emotion. The extracted information is formatted in text format using the MeetingDataExtractor and displayed on the screen.

[0752] Processing flow

[0753] 1. The user queries the robot with a specific keyword or phrase through a voice input device.

[0754] 2. The server uses EmotionEngine to analyze the user's emotions.

[0755] 3. The server uses MeetingDataExtractor to extract the meeting information relevant to the query.

[0756] 4. The extracted information is formatted in text format and displayed on the screen.

[0757] 5. Factory robots will optimize their work based on past meeting information.

[0758] Specific examples

[0759] When a factory worker asks the robot about the "optimal procedure," the robot extracts information about the optimal procedure that was discussed in past meetings and presents it to the worker. For example, the following prompt sentence can be input into the generative AI model:

[0760] Prompt Sentence Examples

[0761] "Please give me some information on the best procedure."

[0762] Based on this prompt, the server extracts relevant procedures and important points from past meeting information and provides them to the worker, allowing the worker to proceed with their work efficiently.

[0763] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0764] Step 1:

[0765] The user queries the robot with specific keywords or phrases through a voice input device.

[0766] Input: User voice input (e.g., "Please give me some information about the best course of action.")

[0767] Output: Audio data

[0768] Specific operation: The user speaks a question into the voice input device, which converts this speech into digital data and sends it to the server.

[0769] Step 2:

[0770] The server uses the Emotion Engine to analyze the user's emotions.

[0771] Input: Audio data

[0772] Output: User's emotion data (e.g., frustrated)

[0773] How it works: The server inputs voice data into the Emotion Engine, which analyzes the user's emotions based on their tone of voice and choice of words. As a result of the analysis, the user's emotional data is generated.

[0774] Step 3:

[0775] The server uses the MeetingDataExtractor to extract the meeting information relevant to the query.

[0776] Input: Keywords and phrases, user sentiment data

[0777] Output: Extracted meeting information (e.g., text data of discussions on optimal procedures)

[0778] Specific operation: Based on the user's inquiry and emotion data, the server uses MeetingDataExtractor to search for related meeting information and extract the relevant information.

[0779] Step 4:

[0780] The extracted information is formatted in text format and displayed on the screen.

[0781] Input: Extracted meeting information

[0782] Output: Textual information (e.g., summary of discussion on optimal procedures)

[0783] Specific operation: The server converts the extracted meeting information into text format and generates data to be displayed on the display. This information is then displayed on the display so that the user can check it.

[0784] Step 5:

[0785] Factory robots optimize their work based on past meeting information.

[0786] Input: Text format information

[0787] Output: Optimized workflow

[0788] How it works: Based on the information displayed on the screen, the factory robot adjusts its work procedures and executes the work in the most optimal way, thereby improving work efficiency.

[0789] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0790] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0791] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0792] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0793] [Second embodiment]

[0794] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0795] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0796] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0797] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0798] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0799] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary-Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0800] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0801] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0802] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0803] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0804] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0805] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0806] "Example 1"

[0807] The system of the present invention has a means for receiving multiple pieces of conference information (video and audio recordings) as input. Specifically, a user inputs recording data of a conference held using a conference tool (e.g., Zoom) into the system.

[0808] "Example 2"

[0809] The system of the present invention has a means for extracting information related to specific content. Specifically, when a user makes a query such as "What did that person say?", the system extracts related content from the input conference information. This extraction is performed using voice recognition technology and natural language processing technology.

[0810] "Example 3"

[0811] The system of the present invention has a means for presenting the extracted information to the user. Specifically, the system presents the extracted information to the user in text format. The system can also present the date, time, participants, and other information of related meetings.

[0812] "Example 4"

[0813] The system of the present invention has a means for extracting information based on a user's inquiry. Specifically, when a user inquires about a specific keyword or phrase, the system extracts information related to that keyword or phrase from the conference information.

[0814] "Example 5"

[0815] The system of the present invention has a means for extracting information across multiple conference records. Specifically, when a user inquires about discussions that took place in multiple conferences, the system extracts related information from the multiple conference records. This allows the user to obtain information across multiple conferences at once, rather than just one conference.

[0816] The processing flow of each embodiment will be described below.

[0817] "Example 1"

[0818] Step 1: Users input recording data of meetings they have held using a conferencing tool (e.g., Zoom) into the system.

[0819] "Example 2"

[0820] Step 1: The user asks a question such as, "What did that person say?"

[0821] Step 2: The system extracts relevant content from the input meeting information using voice recognition and natural language processing technologies.

[0822] "Example 3"

[0823] Step 1: The system presents the extracted information to the user in text format.

[0824] Step 2: Provide relevant meeting dates, times, participants, and other information.

[0825] "Example 4"

[0826] Step 1: User queries for a specific keyword or phrase.

[0827] Step 2: The system extracts information related to the keyword or phrase from the meeting information.

[0828] "Example 5"

[0829] Step 1: A user inquires about discussions that took place across multiple meetings.

[0830] Step 2: The system extracts relevant information from multiple conferences, allowing users to obtain information across multiple conferences at once, rather than just a single conference.

[0831] Example 1

[0832] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0833] Conventional meeting information management systems have had difficulty efficiently extracting necessary information from recorded meeting data and generating summaries. Furthermore, they lacked a means for users to quickly grasp the content of a meeting. This resulted in the problem of requiring a great deal of time and effort when reviewing the content of a meeting later.

[0834] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0835] In this invention, the server includes a means for receiving multiple pieces of conference information (video and audio recordings) as input and converting the audio data into text, a means for inputting a prompt sentence into a generative AI model to generate a summary, and a means for presenting the generated summary to the user. This allows for efficient generation of summaries from recorded conference data, enabling users to quickly understand the content of the conference.

[0836] "Meeting information" refers to video and audio data that records the contents of the meeting.

[0837] "Means for converting voice data to text" refers to technology or devices for analyzing voice data and generating corresponding text data.

[0838] A "generative AI model" refers to an artificial intelligence model that performs natural language processing based on input data to generate summaries and other text.

[0839] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0840] "Means for generating a summary" refers to technology or equipment for creating a summary from input text data using a generative AI model.

[0841] "User presentation means" refers to the techniques and devices used to display or provide the generated summary to the user.

[0842] This invention begins when a user uploads a video file of a conference recorded with a conference tool to the system. The user logs in to the system's web portal, selects the recorded data, and clicks the upload button. The server receives the conference recording data uploaded by the user and temporarily stores it.

[0843] The server then converts the audio portion of the recording into text using the Google Cloud Speech-to-Text API. Specifically, the server extracts the audio track from the recording and sends it to the API, which parses the audio data and returns the corresponding text data.

[0844] The server then inputs a prompt to the generative AI model (e.g., OpenAI's GPT-4). The prompt includes the converted text data and instructions for generating a summary. For example, the prompt might read, "The following text is the content of a one-hour meeting. Please write a summary of this meeting in 500 characters or less."

[0845] The generative AI model generates a summary of the meeting based on this prompt. The server receives the summary and provides it to the user, who can review it through the system's interface. The server then associates the summary with the user's account and saves it for later access.

[0846] As a concrete example, consider the case where a user uploads a recording of a one-hour Zoom meeting to the system. The server first converts the hour's worth of audio data into text using the Google Cloud Speech-to-Text API. Then, it inputs the following prompt to the generative AI model (GPT-4):

[0847] "The text below is from a one-hour meeting. Please write a summary of this meeting in 500 words or less."

[0848] The generative AI model generates a summary of the meeting based on this prompt, and the server provides the generated summary to the user, who can then view it through the system interface.

[0849] The above is a specific embodiment for carrying out the present invention. This system makes it possible to efficiently generate summaries from recorded data of a conference, enabling users to quickly grasp the contents of the conference.

[0850] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0851] Step 1:

[0852] A user uploads a meeting recording.

[0853] A user logs in to the system's web portal, selects the recording data, and clicks the upload button. The input is the meeting recording data selected by the user (e.g., meeting_recording.mp4), and the output is the recording data sent to the server.

[0854] Step 2:

[0855] The server receives the recording data.

[0856] The server receives the conference recording data uploaded by the user and temporarily stores it. The input is the recording data sent by the user, and the output is the recording data saved in the server's storage.

[0857] Step 3:

[0858] The server converts the audio data into text.

[0859] The server converts the audio portion of the recording into text using the Google Cloud Speech-to-Text API. Specifically, the server extracts the audio track from the recording and sends it to the API. The input is the audio track of the recording, and the output is the text data returned by the API.

[0860] Step 4:

[0861] The server inputs a prompt sentence into the generative AI model.

[0862] The server inputs a prompt to the generative AI model (e.g., OpenAI's GPT-4). The prompt includes the converted text data and instructions to generate a summary. The input is the text data and the prompt, and the output is the summary returned by the generative AI model. For example, the prompt might be, "The following text is the content of a one-hour meeting. Please write a summary of this meeting in 500 characters or less."

[0863] Step 5:

[0864] The server provides the generated summary to the user.

[0865] The server receives the summary returned by the generative AI model and provides it to the user. The user can review the summary through the system's interface. The input is the summary returned by the generative AI model, and the output is the summary provided to the user. The server associates the summary with the user's account and stores it for later access by the user.

[0866] (Application example 1)

[0867] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0868] Conventional meeting information management systems simply store recorded meeting data, making it difficult to quickly extract necessary information later. Furthermore, they lack a function to summarize and report the contents of meetings, which means that users have to spend a lot of time reviewing the entire contents of a meeting. This can lead to a decrease in efficiency, especially in situations where quick decision-making is required, such as in factories.

[0869] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0870] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input, extracting information related to specific content, presenting the extracted information to the user, summarizing the extracted information, and reporting the summarized information to the user. This makes it possible to quickly extract necessary information from recorded conference data and summarize and report it.

[0871] "Multiple conference information" refers to video and audio data recorded in multiple conferences.

[0872] "Means of receiving input" refers to the interfaces and processes used to input meeting information into the system.

[0873] "Means for extracting information related to specific content" refers to algorithms or techniques for identifying and extracting information related to specific topics or keywords from meeting information.

[0874] "Means for presenting extracted information to a user" refers to an interface or method for providing extracted information to a user visually or audibly.

[0875] "Means for summarizing extracted information" refers to algorithms or techniques for concisely summarizing extracted information.

[0876] "Means for reporting summarized information to the user" refers to the interface or method for notifying the user of the summarized information.

[0877] A system for implementing this invention receives multiple pieces of conference information (video and audio recordings) as input, extracts information related to specific content, summarizes the extracted information, and reports it to the user. A specific embodiment of this system is described below.

[0878] Hardware and software used

[0879] Hardware:

[0880] Factory robots (e.g. KUKA, Fanuc, etc.)

[0881] Audio Capture Device

[0882] Computer (may be built into the robot)

[0883] software:

[0884] Python

[0885] moviepy (video processing library)

[0886] speech_recognition (speech recognition library)

[0887] transformers (AI models for generating Hugging Face)

[0888] Data processing and calculation

[0889] 1. Extract audio from video:

[0890] The server uses moviepy to extract audio from the conference video and generate a temporary audio file, which is used for subsequent speech recognition processing.

[0891] 2. Speech Recognition:

[0892] The server uses the speech_recognition library to convert the extracted speech into text, which is necessary to handle the contents of the meeting as text information.

[0893] 3. Text Summary:

[0894] The server uses Hugging Face's transformers library to summarize the text using a generative AI model (BART), and this summarized text is reported back to the user.

[0895] 4. Information Presentation and Reporting:

[0896] The server provides an interface for presenting the summarized information to the user, specifically by sending a notification to the user's terminal and displaying the summarized meeting content.

[0897] Specific examples

[0898] Factory managers input the video data of weekly production meetings into a robot. The robot analyzes the video data, summarizes important topics and decisions, and reports them to managers. This saves managers the time of having to review the entire meeting content, allowing them to quickly grasp only the important information.

[0899] Prompt Sentence Examples

[0900] An example of a prompt to input to a generative AI model is as follows:

[0901] "Please summarize the following text:"

[0902] Text extracted from meeting recordings

[0903] This prompt is used to ask the generative AI model to create a summary.

[0904] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0905] Step 1:

[0906] The server receives a conference video file from the user. The input is the conference video file, and the output is the saving of the video file. Specifically, the user uploads the video file recorded using the conference tool to the server.

[0907] Step 2:

[0908] The server uses moviepy to extract audio from conference videos. The input is the conference video file, and the output is an audio file. Specifically, the server reads the video file, separates the audio track, and generates a temporary audio file.

[0909] Step 3:

[0910] The server uses the speech_recognition library to convert the extracted audio into text. The input is an audio file, and the output is text data. Specifically, the server passes the audio file to a speech recognition engine, which converts the audio data into text information.

[0911] Step 4:

[0912] The server uses Hugging Face's transformers library to summarize text using a generative AI model (BART). The input is text data, and the output is summarized text. Specifically, the server inputs text data into the generative AI model and obtains the summarized text.

[0913] Step 5:

[0914] The server provides an interface for reporting the summarized information to the user. The input is the summary text, and the output is a notification to the user terminal. In concrete terms, the server sends the summary text to the user terminal and displays the notification.

[0915] Example 2

[0916] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0917] Conventional meeting information management systems make it difficult for users to quickly and accurately extract information related to specific content. Furthermore, the process of converting audio data into text data, analyzing that text data, and extracting relevant information is complicated, making it difficult to provide appropriate information in response to user inquiries. This makes it difficult to efficiently grasp the content of meetings, hindering work efficiency.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0919] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for converting the audio data into text data, means for analyzing the text data and extracting related information based on a user's inquiry, and means for presenting the extracted information to the user, thereby enabling the user to quickly and accurately extract information related to specific content.

[0920] "Meeting information" refers to video and audio data that records the contents of the meeting.

[0921] "Content-specific information" refers to information related to a particular topic or question that interests a user.

[0922] "Audio data" refers to digital audio files that record statements and discussions made during a meeting.

[0923] "Text data" refers to data obtained by converting voice data into character information.

[0924] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into corresponding text data.

[0925] "Natural language processing technology" refers to the technology of analyzing text data, understanding meaning and context, and extracting information.

[0926] "User Query" means a question or request entered by a User into the System requesting specific information.

[0927] "Means for extracting related information" refers to methods and technologies for searching and extracting related information from text data based on a user's inquiry.

[0928] "Means for presenting information" refers to the methods and techniques used to display extracted information to the user.

[0929] MODE FOR CARRYING OUT THE INVENTION

[0930] This invention relates to a system that receives meeting information (video and audio recordings) as input from a user, extracts information related to specific content, and presents it to the user. A specific embodiment of this system is described below.

[0931] System configuration

[0932] Hardware and software configuration:

[0933] 1. Server:

[0934] The server receives the voice data and converts it into text data using voice recognition technology.

[0935] The server analyzes the converted text data using natural language processing technology and extracts relevant information based on the user's inquiry.

[0936] The server presents the extracted information to the user.

[0937] 2. Terminal:

[0938] A user uses a terminal to upload the conference information to the server.

[0939] The user checks the information provided by the server through the terminal.

[0940] Specific techniques used:

[0941] As speech recognition technology, we use a general speech recognition API (e.g., Google Cloud Speech-to-Text API).

[0942] As a natural language processing technique, we use generative AI models (e.g., OpenAI's GPT-4 model).

[0943] System Operation

[0944] Upload audio data:

[0945] Users upload the audio data of the conference from their devices to the server. The audio data is typically provided in MP3 or WAV format. Users select the file through the system interface and click the upload button.

[0946] Audio data conversion:

[0947] The server receives the uploaded voice data and converts it into text data using voice recognition technology, extracting the spoken content from the voice data as text.

[0948] Analyzing text data:

[0949] The server then analyzes the converted text data using natural language processing technology, specifically using a generative AI model to extract information relevant to the user's inquiry.

[0950] Extracting relevant information:

[0951] The server searches for and extracts relevant comments from text data based on the user's inquiry. For example, in response to a query such as "What did that person say?", it extracts comments from a specific person.

[0952] Providing results:

[0953] The server returns the extracted relevant information to the user, who can check the results through their terminal.

[0954] Specific examples

[0955] If a user uploads audio from a meeting and asks, "What did they say about the progress of the project?", the system will proceed as follows:

[0956] 1. The user opens the browser on the terminal and accesses the system's web page. The user clicks the "Choose File" button and selects an audio file (e.g., "meeting_audio.mp3") from the local disk. Then, the user clicks the "Upload" button.

[0957] 2. When the server receives the voice data, it sends it to the speech recognition API. The API analyzes the voice data and returns the corresponding text data (e.g., "The project is 50% complete."). The server stores this text data.

[0958] 3. The server receives the user's query (e.g., "What did he say about the progress of the project?"). The server inputs the text data and the query into the generative AI model and extracts relevant utterances. For example, the server obtains a result such as, "Regarding the progress of the project, he said: 'Currently, progress is at 50%.'"

[0959] 4. The server sends the extracted information to the user's device. The user can check the results through the system interface. For example, a message like "The project progress was reported as follows: 'Currently, progress is 50%.'" will be displayed on the browser.

[0960] Example prompt sentence:

[0961] "I've uploaded the audio of the meeting. Can you tell me what they said about the project's progress?"

[0962] In this way, the server realizes a system that provides appropriate information in response to user inquiries.

[0963] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0964] Program processing flow

[0965] Step 1: Upload your audio data

[0966] Specific behavior:

[0967] The user opens the browser on the terminal and accesses the system's web page. The user clicks the "Choose File" button and selects an audio file (e.g., "meeting_audio.mp3") from the local disk. Then, the user clicks the "Upload" button.

[0968] input:

[0969] An audio file selected by the user (e.g., "meeting_audio.mp3")

[0970] output:

[0971] Audio files uploaded to the server

[0972] Step 2: Receiving and storing audio data

[0973] Specific behavior:

[0974] The server receives the voice data uploaded by the user and stores it in a specified directory.

[0975] input:

[0976] User-uploaded audio files

[0977] output:

[0978] Audio files stored on the server

[0979] Step 3: Convert audio data to text

[0980] Specific behavior:

[0981] The server sends the stored voice data to a voice recognition API (e.g., a general voice recognition API), which analyzes the voice data and returns the corresponding text data.

[0982] input:

[0983] Audio files stored on the server

[0984] Data processing:

[0985] Convert voice data into text using speech recognition API

[0986] output:

[0987] Text data (e.g., "Project progress is 50%.")

[0988] Step 4: Save the text data

[0989] Specific behavior:

[0990] The server receives the text data returned from the speech recognition API and stores it in a database or file system.

[0991] input:

[0992] Text data returned from the speech recognition API

[0993] output:

[0994] Text data stored on the server

[0995] Step 5: Receiving user inquiries

[0996] Specific behavior:

[0997] A user enters a query through the system interface, for example, "What did they say about the progress of the project?"

[0998] input:

[0999] What the user is asking (e.g., "What did they say about the project status?")

[1000] output:

[1001] User inquiry sent to the server

[1002] Step 6: Analyze text data and extract relevant information

[1003] Specific behavior:

[1004] The server inputs the stored text data and the user's query into a generative AI model (e.g., OpenAI's GPT-4 model) to extract relevant information.

[1005] input:

[1006] Text data stored on the server, user inquiries

[1007] Data Calculation:

[1008] Analyze text data using generative AI models to extract relevant information

[1009] output:

[1010] Extracted relevant information (e.g., "Regarding the project progress, they said: 'Currently, it's 50% complete.'")

[1011] Step 7: Present the extracted information

[1012] Specific behavior:

[1013] The server sends the extracted relevant information to the user's terminal, and the user can check the results through the system interface.

[1014] input:

[1015] Extracted related information

[1016] output:

[1017] Relevant information displayed on the user's device (e.g., "Regarding the project's progress, they said: 'Currently, it's 50% complete.'")

[1018] In this way, the server realizes a system that provides appropriate information in response to user inquiries.

[1019] (Application example 2)

[1020] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1021] With conventional meeting information management systems, it was difficult to quickly and accurately extract specific content when searching for meeting details later. Furthermore, few systems utilized voice recognition or natural language processing technology, making it difficult to provide appropriate answers to user inquiries. Furthermore, they lacked the functionality to extract information across multiple meeting information sources, resulting in low user convenience.

[1022] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to the user, means for converting speech to text using speech recognition technology, and means for generating answers to user questions using natural language processing technology. This allows the user to quickly and accurately search for specific content and extract information across multiple pieces of conference information, improving convenience.

[1023] "Meeting information" refers to video and audio data recorded during a meeting.

[1024] "Information related to specific content" is a part of the conference information related to the content inquired by the user.

[1025] The "means for extracting information" refers to a technique or method for extracting information related to specific content from the conference information.

[1026] "Means for presenting to the user" refers to the techniques or methods for displaying the extracted information to the user.

[1027] "Speech recognition technology" is a technology that converts voice data into text data.

[1028] "Natural language processing technology" is a technology that analyzes text data and generates appropriate answers to user questions.

[1029] A "user question" is the content of an inquiry that a user makes to the system.

[1030] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[1031] The term "means for extracting information across multiple pieces of conference information" refers to a technique or method for extracting information related to specific content from multiple pieces of conference information.

[1032] A system for implementing this invention is configured as follows: The server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to a user, means for converting speech into text using speech recognition technology, and means for generating answers to user questions using natural language processing technology.

[1033] First, the server uses speech recognition technology to convert the audio data of the conference information into text data. For example, the "speech_recognition" library can be used as the speech recognition technology. The server reads the audio file and converts the audio data into text.

[1034] The server then uses natural language processing (NLP) techniques, such as the "pipeline" technique from the "transformers" library, to generate an answer to the user's question. Specifically, the server uses the "deepset / roberta-base-squad2" model to generate an appropriate answer to the user's question.

[1035] When a user wants to search for a specific topic, the server receives the user's question and uses speech recognition and natural language processing technologies to extract relevant information from the meeting information and present it to the user, allowing the user to search for the specific topic quickly and accurately.

[1036] As a concrete example, consider the case where a search is made for "discussions made about the installation of a new production line" during a meeting in a factory. When a user asks, "What was said about the installation of a new production line?", the server extracts relevant content from the meeting information and replies, "The installation of the new production line is scheduled to begin next month, with a budget of 5 million yen."

[1037] Example prompt sentence:

[1038] User: What did they say about setting up a new production line?

[1039] System: Installation of the new production line is scheduled to begin next month, with a budget of 5 million yen.

[1040] In this way, the server can combine speech recognition and natural language processing technologies to generate and present appropriate answers to user questions, enabling users to search for information across multiple conference information sources, improving convenience.

[1041] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1042] Step 1:

[1043] The server receives multiple meeting information (video and audio recordings) as input. Specifically, the server receives audio and video files uploaded by users. This input data is used in subsequent processing steps.

[1044] Step 2:

[1045] The server uses speech recognition technology to convert the received voice data into text data. Specifically, it uses the "speech_recognition" library to read the audio file and convert the voice data into text. The input of this step is voice data, and the output is text data.

[1046] Step 3:

[1047] The server stores the converted text data, which is used in a subsequent natural language processing step, whose input is the text data and whose output is the stored text data.

[1048] Step 4:

[1049] When a user wants to search for something specific, they enter a question. Specifically, the user asks the system, "What did they say about installing a new production line?" The input for this step is the user's question, and the output is the question text.

[1050] Step 5:

[1051] The server uses natural language processing techniques to generate answers to user questions. Specifically, it uses the "pipeline" from the "transformers" library and the "deepset / roberta-base-squad2" model to generate appropriate answers to questions. The input for this step is the question text and stored text data, and the output is the generated answer.

[1052] Step 6:

[1053] The server presents the generated answer to the user. Specifically, it displays the answer on the user's screen. The input of this step is the generated answer, and the output is the answer presented to the user.

[1054] Step 7:

[1055] The user can review the presented answers and ask further questions if necessary. The input of this step is the answers presented to the user, and the output is the user's review.

[1056] In this way, the server can combine speech recognition and natural language processing technologies to generate and present appropriate answers to user questions, enabling users to search for information across multiple conference information sources, improving convenience.

[1057] Example 3

[1058] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1059] Conventional meeting information management systems make it difficult for users to quickly and accurately obtain specific information. They also lack the functionality to extract related information across multiple meeting records, preventing users from obtaining all the information they need at once. Furthermore, they lacked the means to present the extracted information in an easy-to-understand text format, resulting in low user convenience.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1061] In this invention, the server includes means for receiving multiple pieces of conference information as input and extracting information related to specific content, means for extracting information based on user inquiries, means for analyzing the extracted information using a natural language processing engine, means for converting the analyzed information into text format using a generative AI model, and means for presenting the converted text to the user. This allows the user to quickly and accurately obtain specific information and to simultaneously obtain related information across multiple pieces of conference information. Furthermore, presenting the extracted information in an easy-to-understand text format improves user convenience.

[1062] "Meeting information" refers to recorded data such as audio, video, and text related to the meeting.

[1063] "Means for extracting information" refers to techniques and methods for extracting information related to specific content from the meeting information.

[1064] "User Enquiry" refers to keywords or phrases that a user enters into the system to obtain specific information.

[1065] A "natural language processing engine" refers to software that analyzes user inquiries and extracted information and understands their meaning.

[1066] "Generative AI model" refers to an artificial intelligence model that converts analyzed information into natural language text format.

[1067] "Means for converting to text format" refers to the techniques and methods for converting extracted information into a text format that is easy for users to understand.

[1068] "Means for presenting to the user" refers to the technology or method for displaying the converted text on the user's device.

[1069] "Means for extracting information across multiple conference records" refers to techniques and methods for extracting related information from multiple conference records at once.

[1070] "Server" means a computer system that receives user queries and extracts, analyzes, converts, and presents information.

[1071] "Terminal" refers to a device through which a user can enter queries and receive results.

[1072] The present invention relates to a system for extracting conference information based on an inquiry from a user and presenting it in text format. A specific embodiment of this system will be described below.

[1073] First, this system uses the following hardware and software:

[1074] Hardware: Servers, terminals

[1075] Software: Database Management System (DBMS), Natural Language Processing (NLP) engine, Generative AI model

[1076] The server receives inquiries from users. When the user inputs specific keywords or phrases, the server sends the information to an NLP engine. The NLP engine extracts relevant meeting information from a database based on the input keywords or phrases. The extracted information is converted into text using a generative AI model. The server then sends the generated text to the user's device.

[1077] As a concrete example, the following processing occurs when a user asks, "Who are the participants in the next meeting?"

[1078] 1. The user types into the device interface, "Who's in my next meeting?"

[1079] 2. The device sends this query to the server.

[1080] 3. The server receives the query and sends it to the NLP engine.

[1081] 4. The NLP engine analyzes the keywords "next meeting" and "participants" and extracts relevant meeting information from the database.

[1082] 5. The server sends the extracted information to the generative AI model.

[1083] 6. The generative AI model generates the text, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1084] 7. The server sends the generated text to the terminal.

[1085] 8. The device displays to the user, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1086] In this way, the user can easily obtain the information he or she needs.

[1087] An example of a prompt sentence might be:

[1088] Who will be at your next meeting?

[1089] After entering this prompt, the system will present the user with a list of participants for the next meeting in text format. For example, the result might look like this: "The participants for the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1090] This system allows users to quickly and accurately obtain specific information, and to simultaneously obtain related information across multiple conference information. Furthermore, by presenting the extracted information in an easy-to-understand text format, user convenience is improved. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[1091] Step 1:

[1092] The user enters a query.

[1093] The user uses the device's interface to enter a query containing specific keywords or phrases, for example, "Who are the participants in the next meeting?" The entered data is temporarily stored in the device's memory.

[1094] Step 2:

[1095] The terminal sends a query to the server.

[1096] The terminal sends the query entered by the user to the server using a communication protocol such as an HTTP request. The input data is transferred to the server via the network.

[1097] Step 3:

[1098] The server receives the query and sends it to the NLP engine.

[1099] The server analyzes queries received from the device and sends them to the NLP engine. The server analyzes the received data and converts it into a format that the NLP engine can understand. The converted data is then passed to the NLP engine.

[1100] Step 4:

[1101] The NLP engine analyzes keywords and extracts information from the database.

[1102] The NLP engine analyzes the data received from the server and extracts specific keywords and phrases. For example, it searches the database based on keywords like "next meeting" or "participants." It then extracts relevant meeting information from the database and returns that information to the server.

[1103] Step 5:

[1104] The server sends the extracted information to the generative AI model.

[1105] The server sends the extracted information received from the NLP engine to the generative AI model. The server converts the extracted data into a format that the generative AI model can understand and passes it to the generative AI model.

[1106] Step 6:

[1107] The generative AI model converts it into text format.

[1108] The generative AI model converts the information received from the server into a text format that is easy for the user to understand. For example, it generates text such as, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato." The generated text is then returned to the server.

[1109] Step 7:

[1110] The server sends the generated text to the terminal.

[1111] The server sends the text received from the generative AI model to the device, again using a communication protocol such as an HTTP request. The generated text is then transferred to the device via the network.

[1112] Step 8:

[1113] The terminal displays the results to the user.

[1114] The terminal displays the text received from the server to the user. The user can check the results on the terminal screen. For example, the result may read, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1115] (Application example 3)

[1116] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1117] There is a need to efficiently manage information on multiple meetings and discussions within a factory and quickly extract and present information related to specific keywords. However, conventional systems have had issues with cross-sectional searches of multiple meeting information, making it difficult to quickly obtain the necessary information. Furthermore, they lacked the functionality to present the extracted information along with the date, time, and participant information of related meetings, making it difficult for users to comprehensively grasp the information they need.

[1118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1119] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to a user, means for presenting the extracted information along with the date and time of related conferences and attendee information, means for extracting information across multiple pieces of conference information, means for a user to input specific keywords or phrases, and means for searching for information based on the keywords or phrases input by the user. This allows a user to efficiently manage information on multiple conferences and meetings within a factory and quickly obtain information related to specific keywords.

[1120] "Meeting information" refers to video and audio data recorded during meetings and discussions.

[1121] The "means for extracting information" refers to a method or device for extracting information related to specific content from input conference information.

[1122] "Means for presenting to the user" refers to a method or device for visually or audibly presenting the extracted information to the user.

[1123] The "means for presenting the date and time and participant information" refers to a method or device for providing the user with the date and time and participant information of the related conference together with the extracted information.

[1124] The "means for extracting information across multiple conferences" refers to a method or device for extracting related information collectively from multiple conference information.

[1125] A "means for entering keywords or phrases" is a method or device that allows a user to enter specific keywords or phrases into the system.

[1126] "Means for searching information" refers to a method or device for retrieving relevant information from conference information based on keywords or phrases entered by the user.

[1127] The system for implementing this invention aims to efficiently manage information on multiple meetings and conferences within a factory and quickly extract and present information related to specific keywords. A specific embodiment of this system will be described below.

[1128] System configuration

[1129] The system consists of the following major components:

[1130] 1. Database server: Manages the database for storing conference information (video and audio recordings).

[1131] 2. Search server: Searches conference information and extracts relevant information based on keywords and phrases provided by the user.

[1132] 3. User Interface: Provides an interface for users to enter keywords or phrases and view search results.

[1133] Hardware and software used

[1134] Hardware: Factory robots, servers

[1135] Software: SQLite (database management system), Python (programming language)

[1136] Data processing and calculation flow

[1137] 1. Database connection: The database server uses SQLite to store conference information, including video and audio recordings, conference dates and times, and participant information.

[1138] 2. Information extraction: The search server extracts relevant information from the conference information based on the keywords or phrases entered by the user, specifically by using SQL queries to search for detailed conference information.

[1139] 3. Result presentation: The extracted information is presented to the user through a user interface, including the meeting date, time, participants, and details.

[1140] Specific examples

[1141] For example, if a factory manager wants to search for past meeting information using the keyword "quality control," he or she would perform the following operations.

[1142] 1. Enter the keyword "quality control" into the user interface.

[1143] 2. The search server connects to the database server and searches for relevant conference information.

[1144] 3. The extracted information, along with the meeting date, time, and participant information, is displayed in the user interface.

[1145] Prompt Sentence Examples

[1146] "Find information about quality control discussed at past meetings."

[1147] By inputting this prompt into a generative AI model, relevant meeting information can be quickly retrieved.

[1148] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1149] Step 1:

[1150] A user uses a terminal to enter a particular keyword or phrase into a user interface.

[1151] Input: Keyword or phrase (e.g., "quality control")

[1152] Output: The keyword or phrase you entered

[1153] Step 2:

[1154] The terminal transmits the entered keywords or phrases to a search server.

[1155] Input: Keywords or phrases entered by the user

[1156] Output: Keywords or phrases sent to the search server

[1157] Step 3:

[1158] The search server connects to a database server and searches for relevant conference information based on the received keywords or phrases.

[1159] Input: Submitted keyword or phrase

[1160] Output: Search results using SQL queries (related conference information)

[1161] Step 4:

[1162] The database server searches the conference information and extracts relevant information based on the SQL query from the search server.

[1163] Input: SQL query

[1164] Output: Extracted meeting information (meeting date, time, participants, details)

[1165] Step 5:

[1166] The search server processes the extracted information received from the database server and converts it into a format for presentation to the user.

[1167] Input: Extracted meeting information

[1168] Output: Information transformed into a format for presentation to the user

[1169] Step 6:

[1170] The search server transmits the converted information to the terminal.

[1171] Input: Formatted information

[1172] Output: Information sent to the terminal

[1173] Step 7:

[1174] The terminal displays the received information on a user interface and presents it to the user.

[1175] Input: Received information

[1176] Output: Information displayed in the user interface (meeting date, time, participants, details)

[1177] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1178] "Example 1"

[1179] In one embodiment of the present invention, a recording of a meeting held by a user using a conferencing tool (e.g., Zoom) is input to the system. The system includes an emotion engine that recognizes emotions from the user's voice.

[1180] "Example 2"

[1181] When a user asks a question such as "What did that person say?", the system extracts relevant content from the input meeting information. This extraction is done using speech recognition and natural language processing technologies. Furthermore, the emotion engine recognizes emotions from the user's voice and extracts information based on those emotions.

[1182] "Example 3"

[1183] The system presents the extracted information to the user in text format, along with relevant meeting dates, participants, and other information. The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[1184] "Example 4"

[1185] When users query for specific keywords or phrases, the system extracts information related to those keywords or phrases from the meeting information.The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[1186] "Example 5"

[1187] When a user inquires about discussions that took place in multiple meetings, the system extracts relevant information from multiple meeting records. This allows users to obtain information across multiple meetings at once, rather than just one. The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[1188] The processing flow of each embodiment will be described below.

[1189] "Example 1"

[1190] Step 1: Users input recording data of meetings they have held using a conferencing tool (e.g., Zoom) into the system.

[1191] Step 2: The system activates an emotion engine that recognizes emotions from the user's voice.

[1192] Step 1: The user asks a question such as, "What did that person say?"

[1193] Step 2: The system extracts relevant content from the input meeting information using voice recognition and natural language processing technologies.

[1194] Step 3: The emotion engine recognizes emotions from the user's voice and extracts information based on those emotions.

[1195] "Example 3"

[1196] Step 1: The system presents the extracted information to the user in text format.

[1197] Step 2: Provide relevant meeting dates, times, participants, and other information.

[1198] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[1199] "Example 4"

[1200] Step 1: User queries for a specific keyword or phrase.

[1201] Step 2: The system extracts information related to the keyword or phrase from the meeting information.

[1202] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[1203] "Example 5"

[1204] Step 1: A user inquires about discussions that took place across multiple meetings.

[1205] Step 2: The system extracts relevant information from multiple conferences, allowing users to obtain information across multiple conferences at once, rather than just a single conference.

[1206] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[1207] Example 1

[1208] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1209] Conventional meeting recording systems simply store the contents of the meeting, making it difficult to grasp the emotional state of the participants during the meeting. This makes it difficult to properly evaluate the effectiveness of the meeting or the reactions of the participants, and makes it difficult to obtain specific feedback to improve the quality of the meeting.

[1210] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1211] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input, separating the video data from the audio data, means for analyzing the audio data and recognizing emotions, means for storing the recognized emotions in a database, and means for presenting the stored emotion analysis results to the user, thereby enabling automatic analysis and visualization of the emotional states of participants during a conference.

[1212] "Meeting information" refers to video and audio data recorded during a meeting.

[1213] "Video data" refers to data that includes video information from a conference.

[1214] "Audio data" refers to data containing audio information during a conference.

[1215] "Means for separating" refers to the technical means for extracting video data and audio data separately.

[1216] "Means for analyzing" refers to the technical means for analyzing the audio data and extracting specific information.

[1217] "Means for recognizing emotions" refers to technical means for identifying emotional states from audio data.

[1218] "Database" refers to a data storage system for storing analytical results and other information.

[1219] "Presentation Means" refers to the technical means by which information is displayed or presented to a user.

[1220] "User" refers to any individual or organization that uses the System.

[1221] "Query" refers to a request for information made by a user to a system.

[1222] "Means for extracting information across multiple conferences" refers to technical means for comparing and analyzing multiple conference information to extract common or related information.

[1223] MODE FOR CARRYING OUT THE INVENTION

[1224] This invention is a system that receives multiple pieces of conference information (video and audio recordings) as input, separates the video data from the audio data, analyzes the audio data to recognize emotions, and presents the results to the user.

[1225] Hardware and software used

[1226] The server uses the following hardware and software:

[1227] Hardware: powerful processor, sufficient memory, storage devices, and network interfaces

[1228] Software: FFmpeg (video processing library), Google Cloud Speech-to-Text API, IBM Watson sentiment analysis API, database management system (e.g., MySQL)

[1229] Data processing and calculation

[1230] The server receives the meeting recording data uploaded by the user and performs the following processes:

[1231] 1. Separating video and audio data: Use FFmpeg to separate video and audio data. The audio data is converted to WAV format.

[1232] 2. Analyze voice data: Use Google Cloud Speech-to-Text API and IBM Watson sentiment analysis API to analyze voice data and identify the user's emotional state.

[1233] 3. Save to database: The analysis results are saved to a database, including the meeting timestamp, emotion type, and emotion intensity.

[1234] 4. Presentation to the user: Based on the saved sentiment analysis results, feedback is provided to the user. The feedback is generated as a report visualizing changes in sentiment and displayed on the user's dashboard.

[1235] Specific examples

[1236] As a concrete example, consider a scenario in which a user uploads recording data of a meeting held using a conferencing tool to the system. When the user uploads the recording data of the meeting to the system, the server receives the data and inputs the voice data into the emotion engine. The emotion engine recognizes the user's emotion from the voice data and stores the result in a database. Finally, the user can receive a report visualizing the changes in emotion during the meeting.

[1237] Prompt Sentence Examples

[1238] Users upload recordings of meetings held using a conferencing tool to the system. The system analyzes the audio data and uses an emotion engine to recognize the user's emotions. The analysis results are stored in a database and a report is generated that visualizes changes in emotions.

[1239] In this way, it is possible to specifically explain how the server, terminal, and user work together to utilize the system.

[1240] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1241] Step 1:

[1242] A user uploads a meeting recording.

[1243] The user accesses the system's web interface, selects the conference recording data, and clicks the upload button. Once the upload is complete, the server receives the data.

[1244] Input: Meeting recording data (video and audio recording)

[1245] Output: Meeting recording data saved on the server

[1246] Step 2:

[1247] The server receives the recorded data and separates the video data from the audio data.

[1248] The server temporarily stores the received recording data and calls the FFmpeg library to separate the video data from the audio data. The audio data is saved as a WAV file.

[1249] Input: Meeting recording data stored on the server

[1250] Output: Separated video and audio data (WAV format)

[1251] Step 3:

[1252] The server inputs the voice data into the emotion engine.

[1253] The server sends the stored voice data to the emotion engine, which receives the voice data and starts analyzing it.

[1254] Input: Separated audio data (WAV format)

[1255] Output: Voice data input to the emotion engine

[1256] Step 4:

[1257] The emotion engine analyzes voice data and recognizes emotions.

[1258] The emotion engine analyzes the audio data and identifies the user's emotion at each timestamp, for example, recognizing that the user is feeling "joyful" five minutes into the meeting.

[1259] Input: Speech data input to the emotion engine

[1260] Output: Parsed emotion data (timestamp, emotion type, emotion intensity)

[1261] Step 5:

[1262] The server stores the analysis results in a database.

[1263] The server stores the analysis results received from the emotion engine in a database, including the meeting timestamp, emotion type, and emotion intensity.

[1264] Input: Parsed emotion data

[1265] Output: Sentiment analysis results stored in a database

[1266] Step 6:

[1267] The server provides feedback on the emotion analysis results to the user.

[1268] The server provides feedback to the user based on the emotion analysis results stored in the database. The feedback is generated as a report visualizing changes in emotion and displayed on the user's dashboard.

[1269] Input: Sentiment analysis results stored in the database

[1270] Output: Sentiment analysis results presented to the user (visual report)

[1271] (Application example 1)

[1272] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1273] With conventional conferencing systems, it was difficult to grasp the emotions of participants in real time during a meeting, and it was particularly difficult to detect abnormal emotional changes (e.g., anger, fear, stress) and take appropriate measures. Furthermore, it was not possible to take user emotions into consideration when extracting information related to specific content from recorded meeting data, making it impossible to improve the quality and safety of meetings.

[1274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1275] In this invention, the server includes a means for receiving multiple pieces of conference information (video and audio recordings) as input, extracting information related to specific content, presenting the extracted information to the user, recognizing emotions from the user's voice, and issuing an alert if the recognized emotion is abnormal. This makes it possible to grasp the emotions of participants during a conference in real time, detect abnormal emotional changes, and take appropriate action. Furthermore, the user's emotions can be taken into consideration when extracting information related to specific content from the conference recording data, thereby improving the quality and safety of the conference.

[1276] "Meeting information" refers to video and audio data recorded during a meeting.

[1277] "Information related to a specific content" refers to information extracted from the conference information that is related to a specific topic or theme that is of interest to the user.

[1278] The term "extraction means" refers to a technical means for extracting information related to specific content from the conference information.

[1279] "Presentation means" refers to the technical means by which the extracted information is presented to the user visually or audibly.

[1280] "Means for recognizing emotions" refers to technical means for analyzing emotions from a user's voice data and identifying a specific emotional state.

[1281] "Means for issuing an alert" refers to a technical means for issuing a warning to a user or administrator when a recognized emotion is abnormal.

[1282] The system for implementing this invention receives multiple meeting information (video and audio recordings) as input, extracts information related to specific content, and presents it to the user. It also has the function of recognizing emotions from the user's voice and issuing an alert if the recognized emotion is abnormal.

[1283] Hardware and software used

[1284] Hardware:

[1285] server

[1286] Smartphone or smart glasses

[1287] microphone

[1288] software:

[1289] pyaudio library (audio recording)

[1290] wave library (storing audio data)

[1291] emotion_recognition library (emotion recognition)

[1292] System processing overview

[1293] Audio recording

[1294] The device (smartphone or smart glasses) records the audio during the meeting. It uses the pyaudio library to capture audio data in real time and save it in WAV format.

[1295] emotion recognition

[1296] The server receives the recorded audio data and analyzes the emotions using the emotion_recognition library, which analyzes the audio data and identifies the user's emotional state.

[1297] Information Extraction and Presentation

[1298] The server extracts information related to a specific content from the conference information, and the extracted information is presented to the user visually or audibly.

[1299] Alert occurs

[1300] The server issues an alert if the recognized emotion is abnormal (e.g., anger, fear, stress), which is then notified to the user or administrator.

[1301] Specific examples

[1302] For example, if a participant suddenly becomes angry during a meeting, the device will record the audio and send it to the server. The server will then analyze the recorded audio and recognize the emotion of "anger." If the recognized emotion is abnormal, the server will send an alert to the administrator and prompt them to take appropriate action.

[1303] Prompt Sentence Examples

[1304] Analyze the audio data recorded during the meeting and recognize the emotions of the participants. If emotions such as "anger," "fear," and "stress" are detected, an alert will be issued.

[1305] In this way, it is possible to grasp the emotions of participants in a meeting in real time, detect abnormal emotional changes, and take appropriate action.In addition, when extracting information related to specific content from recorded meeting data, user emotions can be taken into consideration, improving the quality and safety of meetings.

[1306] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1307] Step 1:

[1308] The terminal records audio during a conference. It uses the pyaudio library to capture audio data in real time and save it in WAV format. The input is the audio signal from the microphone, and the output is a WAV audio file. Specifically, the terminal reads the audio data at regular intervals and saves it in a buffer.

[1309] Step 2:

[1310] The terminal sends the recorded audio data to the server. The input is a WAV format audio file, and the output is the transfer of audio data to the server. Specifically, the terminal uploads the audio file to the server via the network.

[1311] Step 3:

[1312] The server analyzes the received audio data and recognizes emotions. It uses the emotion_recognition library to analyze the audio data and identify the user's emotional state. The input is a WAV audio file, and the output is the recognized emotion (e.g., anger, fear, stress). Specifically, the server inputs the audio data into the library and runs the emotion recognition model.

[1313] Step 4:

[1314] The server determines whether the recognized emotion is abnormal. The input is the recognized emotion, and the output is the abnormal emotion determination result. Specifically, the server compares the recognized emotion with a predefined list of abnormal emotions, and if there is a match, it determines that the emotion is abnormal.

[1315] Step 5:

[1316] The server issues an alert when an abnormal emotion is detected. The input is the abnormal emotion judgment result, and the output is an alert notification. Specifically, the server sends an email or push notification to the administrator to inform them that an abnormal emotion has been detected.

[1317] Step 6:

[1318] The server extracts information related to specific content from the meeting information. The input is the meeting information (video and audio recordings), and the output is the extracted information. Specifically, the server uses natural language processing technology to extract information related to specific topics or themes that interest the user from the meeting information.

[1319] Step 7:

[1320] The server presents the extracted information to the user. The input is the extracted information, and the output is the presentation of the information to the user. In concrete terms, the server displays the extracted information through an interface for providing it to the user visually or audibly.

[1321] Example 2

[1322] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1323] Conventional meeting information management systems have made it difficult for users to quickly search for specific content, especially when extracting information from audio data. Furthermore, they do not provide information that takes into account the user's emotions, which means they are unable to provide information that meets the user's needs.

[1324] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a plurality of pieces of conference information as input and extracting information related to specific content, means for presenting the extracted information to the user, means for receiving a voice inquiry from the user and converting the voice into text, means for extracting related information based on the converted text, and means for recognizing emotions from the user's voice and filtering information based on the emotions. This enables the user to quickly and accurately obtain information related to specific content, and further enables information to be provided according to the user's emotions.

[1325] "Meeting information" refers to data such as audio, video, and text recorded during a meeting.

[1326] "Information related to specific content" refers to information about specific topics or comments within the meeting information that is extracted based on a user's inquiry.

[1327] The term "extraction means" refers to a technique or method for extracting information related to specific content from the conference information.

[1328] "Means of presentation to the user" refers to the technology or method used to display or audibly convey the extracted information to the user.

[1329] "Voice inquiry" refers to a question or request made by a user to a system using voice.

[1330] "Means for converting speech to text" refers to the technology or method for converting speech data into text data.

[1331] "Means for extracting relevant information" refers to techniques and methods for extracting information relevant to the user's inquiry from the converted text data.

[1332] "Means for recognizing emotions" refers to technologies and methods for analyzing emotions from a user's voice and identifying those emotions.

[1333] "Information filtering means" refers to techniques and methods for filtering extracted information in a way that is most relevant to the user based on perceived emotions.

[1334] MODE FOR CARRYING OUT THE INVENTION

[1335] The present invention provides a system that allows a user to quickly and accurately search for specific content from conference information and provides information that corresponds to the user's emotions. A specific embodiment of this system will be described below.

[1336] Hardware and software used

[1337] Hardware: Servers, devices (PCs, smartphones, etc.)

[1338] Software: speech recognition software (e.g., Google Speech-to-Text API), natural language processing software (e.g., OpenAI GPT-4), emotion recognition engines (e.g., IBM Watson Tone Analyzer)

[1339] System Operation

[1340] 1. User Inquiries

[1341] The user uses the device to make a voice inquiry such as, "What did that person say?" For example, the user speaks into the microphone of a smartphone.

[1342] 2. Sending audio data

[1343] The device sends the user's voice data to a server, and the voice data recorded by the smartphone is uploaded to the server via the Internet.

[1344] 3. Voice Recognition

[1345] The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the voice data into text data. The server receives the voice data and calls the Google Speech-to-Text API to convert it from voice to text.

[1346] 4. Natural Language Processing

[1347] The server inputs the converted text data into natural language processing software (e.g., OpenAI GPT-4) to extract relevant content from the meeting information.The server inputs the text data into OpenAI GPT-4 to extract relevant information for the question, "What did that person say?"

[1348] 5. Emotion recognition

[1349] The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the user's voice. The server inputs the voice data into IBM Watson Tone Analyzer and recognizes that the user's emotion is "confused."

[1350] 6. Information Extraction

[1351] The server then filters the information extracted by the natural language processing software based on the recognized emotion and selects the most relevant information for the user.The server selects the clearest and most specific answer based on the emotion "confused."

[1352] 7. Submitting the results

[1353] The server sends the selected information to the device and provides feedback to the user by displaying or speaking. The server sends the selected text to the smartphone, which displays the text on the screen or reads it aloud.

[1354] Specific examples

[1355] For example, if a user asks a question to a terminal during a meeting, such as "What did that person say?", the following specific actions are performed.

[1356] 1. The user speaks into the smartphone microphone, "What did that person say?"

[1357] 2. The smartphone uploads the recorded audio data to a server via the Internet.

[1358] 3. The server receives the voice data and calls the Google Speech-to-Text API to convert it from voice to text.

[1359] 4. The server inputs the text data into OpenAI GPT-4 and extracts relevant information for the question, "What did that person say?"

[1360] 5. The server inputs the voice data into IBM Watson Tone Analyzer and determines that the user's emotion is "confusion."

[1361] 6. The server selects the most clear and specific answer based on the emotion of "confused."

[1362] 7. The server sends the selected text to the smartphone, which displays it on the screen or reads it aloud.

[1363] Prompt Sentence Examples

[1364] An example of a prompt sentence to input to the generative AI model is as follows:

[1365] When a user asks, "What did that person say?", extract relevant utterances from the meeting text data. Provide a clear answer, especially if the user's emotion is "confused."

[1366] Using this prompt, the generative AI model can generate an appropriate answer to the user's question.

[1367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1368] Step 1:

[1369] The user makes a voice inquiry to the device. Specifically, the user speaks into the smartphone microphone, saying, "What did that person say?" The input is the user's voice data, and the output is an audio file recorded on the device.

[1370] Step 2:

[1371] The device sends the recorded audio data to a server. Specifically, the audio data recorded by the smartphone is uploaded to the server via the Internet. The input is the audio file recorded on the device, and the output is the audio data sent to the server.

[1372] Step 3:

[1373] The server uses speech recognition software to convert the voice data into text data. Specifically, the server calls the Google Speech-to-Text API to perform "voice-to-text" conversion. The input is the voice data sent to the server, and the output is the converted text data.

[1374] Step 4:

[1375] The server inputs the converted text data into natural language processing software to extract relevant content from the meeting information. Specifically, the server uses OpenAI GPT-4 to extract relevant information for the question, "What did that person say?" The input is the converted text data, and the output is the extracted relevant information.

[1376] Step 5:

[1377] The server uses an emotion recognition engine to recognize emotions from the user's voice. Specifically, the server uses IBM Watson Tone Analyzer to recognize that the user's emotion is "confused." The input is the voice data sent to the server, and the output is the recognized emotion data.

[1378] Step 6:

[1379] The server filters the information extracted by the natural language processing software based on the recognized emotion and selects the most relevant information for the user. Specifically, the server selects the clearest and most specific answer based on the emotion "confused." The input is the extracted relevant information and the recognized emotion data, and the output is the selected information.

[1380] Step 7:

[1381] The server sends the selected information to the device and provides feedback to the user by display or voice. Specifically, the server sends the selected text to the smartphone, which then displays the text on the screen or reads it aloud. The input is the selected information, and the output is the information presented to the user.

[1382] (Application example 2)

[1383] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1384] In autonomous vehicles, there is a need to be able to review conversations that took place in the car later, but current systems do not adequately provide such functionality. Furthermore, there is a demand for more appropriate information provision by providing information based on the user's emotions. This calls for a system that can search for the content of conversations in the car and provide information based on emotions.

[1385] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1386] In this invention, the server includes: means for accepting a user's inquiry; means for extracting information related to the inquiry from video data or audio data representing multiple conferences or conversations using a prompt sentence instructing the extraction of information related to the inquiry from video data or audio data representing multiple conferences or conversations and a generative AI model; and means for providing the extracted information related to the inquiry to the user. The server further includes means for recognizing a user's emotion using an emotion engine, and the means for providing to the user provides the extracted information related to the inquiry to the user using a prompt sentence instructing the extraction of information related to the inquiry to the user based on the recognized user's emotion and the generative AI model. The multiple conferences or conversations took place in a vehicle. This makes it possible to later check the conferences or conversations that took place in the vehicle, and to provide appropriate information based on the user's emotion.

[1387] "Meeting information" refers to the contents of a meeting recorded in the form of audio, video, or the like.

[1388] "Information related to specific content" refers to a specific part of the meeting information that is extracted based on a user's inquiry or search conditions.

[1389] "Voice recognition technology" refers to technology that converts voice data into text data.

[1390] "Texting" refers to the process of converting non-text data, such as audio or video, into written information.

[1391] "User inquiry" refers to a question or search request made by a user to the system.

[1392] An "emotion engine" is a technology that analyzes emotions from a user's voice or text and provides appropriate information based on those emotions.

[1393] "In-vehicle conversations" refers to voice communications conducted by drivers and passengers inside an autonomous vehicle.

[1394] "Search" refers to the process of finding relevant information from a database or recorded information based on specific criteria.

[1395] "Providing information" means presenting appropriate answers and related information to users' inquiries.

[1396] A system for implementing this invention has the following configuration. First, conversations taking place inside the vehicle are collected using a microphone installed inside the vehicle. The collected voice data is sent to an on-board computer and converted into text data using voice recognition technology. This voice recognition technology uses the SpeechRecognition library.

[1397] The converted text data is then analyzed using natural language processing (NLP) techniques, using the Transformers library and the deepset / roberta-base-squad2 model. When a user issues a query to the system, NLP techniques are used to search for and extract relevant information.

[1398] Furthermore, the system analyzes the user's emotions using an emotion engine. The TextBlob library is used for this emotion analysis. Based on the results of the emotion analysis, the system provides appropriate information to the user.

[1399] As a specific example, if a user asks, "We were talking about what to eat at the next rest stop. What did they say?" the system will search for relevant content from multiple meetings or conversations and provide relevant information.

[1400] An example of a prompt is as follows:

[1401] After inputting video or audio data representing multiple meetings or conversations, the system may ask, "User question: We were talking about what to eat at the next rest stop. What did they say?" Extract information relevant to the inquiry from video or audio data representing multiple meetings or conversations.

[1402] Output of the generative AI model: The conversation went something like, "Let's eat ramen at the next rest stop."

[1403] In this way, multiple conversations that took place in the car can be reviewed later. The server uses a generative AI model and prompts that instruct the server to provide information related to the extracted query to the user based on the recognized user's sentiment. Specifically, if the user's sentiment is positive, the system provides more detailed information, and if it is negative, it provides concise information. This makes it possible to provide appropriate information based on the user's sentiment.

[1404] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1405] Step 1:

[1406] A microphone installed inside the vehicle collects video or audio data representing multiple conversations taking place inside the vehicle.

[1407] Input: In-car video data or audio data

[1408] Output: Collected video or audio data

[1409] How it works: A microphone collects audio from inside the car in real time and transmits it to the onboard computer.

[1410] Step 2:

[1411] The onboard computer converts the collected voice data into text data using voice recognition technology.

[1412] Input: Collected audio data

[1413] Output: Text data

[1414] Specific operation: The vehicle's computer uses the SpeechRecognition library to convert voice data into text data.

[1415] Step 3:

[1416] The onboard computer analyzes the converted text data using natural language processing technology.

[1417] Input: Text data

[1418] Output: Analysis results

[1419] What it does: The onboard computer uses the Transformers library to parse the text data.

[1420] Step 4:

[1421] The user queries the system.

[1422] Input: User query

[1423] Output: Inquiry details

[1424] Specific action: The user asks the system a question via voice or text.

[1425] Step 5:

[1426] A server or on-board computer searches and extracts relevant information based on the user's query.

[1427] Input: Inquiry details, analysis results

[1428] Output: Search results

[1429] Specific behavior: The server or on-board computer uses a prompt statement that instructs the extraction of information related to the inquiry from the text data and a generative AI model to search for and extract information related to the inquiry content from the text data.

[1430] Step 6:

[1431] The server or on-board computer uses an emotion engine to analyze the user's emotions.

[1432] Input: User voice or text

[1433] Output: Emotion analysis results

[1434] Specific operation: The server or on-board computer uses the TextBlob library to analyze emotions from the user's voice or text.

[1435] Step 7:

[1436] The server or the in-vehicle computer provides appropriate information to the user based on the emotion analysis results.

[1437] Input: Search results, sentiment analysis results

[1438] Output: Information provided

[1439] Specific operation: The server or on-board computer uses prompts based on the results of emotion analysis and a generative AI model to provide the user with appropriate information in voice or text.

[1440] Example 3

[1441] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1442] Conventional meeting information management systems make it difficult for users to quickly and accurately obtain specific information. Furthermore, they do not provide information based on the user's emotions, which results in a poor user experience. Furthermore, they lack the functionality to extract related information across multiple meeting information, making it difficult for users to obtain the information they need all at once.

[1443] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1444] In this invention, the server includes means for receiving multiple pieces of conference information as input and extracting information related to specific content, means for presenting the extracted information to the user, means for extracting information based on a user inquiry, means for recognizing the user's emotions and extracting information based on the emotions, and means for presenting the extracted information to the user in text format. This allows the user to quickly and accurately obtain the information they need, and also makes it possible to provide information according to the user's emotions. Furthermore, since related information can be extracted across multiple pieces of conference information, the user can obtain the information they need all at once.

[1445] "Meeting information" refers to data such as audio, video, and text recorded during a meeting.

[1446] "Means for extracting information" refers to techniques and methods for extracting information related to specific content from the meeting information.

[1447] "Means of presentation to the user" refers to the techniques and methods for providing the extracted information to the user visually or audibly.

[1448] "User Inquiry" refers to a question or request made by a user to the system to obtain specific information.

[1449] "Means of emotion recognition" refers to technologies and methods for analyzing and identifying emotions from user input and behavior.

[1450] "Text format" refers to a format in which information is expressed as a string of characters.

[1451] "Means for extracting information across multiple sources" refers to techniques and methods for extracting related information from multiple conference information sources.

[1452] MODE FOR CARRYING OUT THE INVENTION

[1453] The system of the present invention receives multiple pieces of conference information as input, extracts information related to specific content, and presents it to the user. This system has the function of extracting information based on user inquiries and further recognizing the user's emotions and extracting information based on those emotions. Specific embodiments are described below.

[1454] Hardware and software used

[1455] 1. Server

[1456] The server is connected to a database (such as MySQL) that stores conference information.

[1457] The server runs a Python script to extract information from the database.

[1458] The server uses an emotion engine such as IBM Watson Tone Analyzer for emotion recognition.

[1459] 2. Terminal

[1460] Users access the system using devices such as PCs and smartphones.

[1461] The terminal provides a user interface and accepts inquiries from the user.

[1462] Data processing and calculation

[1463] 1. Information Extraction

[1464] The server receives user queries and executes Python scripts to send queries to a MySQL database.

[1465] The server extracts information related to specific keywords or phrases from a database.

[1466] Example: In response to the query "When is the next meeting?", the date and time of the next meeting is retrieved from the database.

[1467] 2. Emotional Recognition

[1468] The server sends the user's input text to the IBM Watson Tone Analyzer to analyze the user's emotions.

[1469] The server receives the analysis results and recognizes that the user's emotions are "interest" or "concern."

[1470] 3. Presentation of Information

[1471] The server formats the extracted information in text format and returns it to the user.

[1472] Example: Generate a message saying "The next meeting will be held on October 15, 2023 at 2:00 PM" and send it to the user's device.

[1473] Specific examples

[1474] When the user asks, "When is the next meeting?" the server responds, "The next meeting will be on October 15, 2023 at 2:00 p.m."

[1475] When the user asks, "What were the main discussions in the last three meetings?" the server responds, "The main discussions in the last three meetings were: 1. New product development 2. Marketing strategy 3. Budget review."

[1476] Prompt Sentence Examples

[1477] "When is the next meeting?"

[1478] "What were the main discussions at the last three meetings?"

[1479] This system is designed to provide information quickly and accurately in response to user inquiries. In addition, by using an emotion engine, it is possible to provide appropriate information according to the user's emotions. Furthermore, since it is possible to extract related information across multiple conference information, the user can obtain the information they need at once. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[1480] Step 1:

[1481] User inquiries

[1482] The user accesses the system interface using a device (PC or smartphone), enters a specific keyword or phrase in the text box, and clicks the send button.

[1483] Input: A keyword or phrase typed by the user (e.g., "When is our next meeting?")

[1484] Output: The user's query is sent to the server.

[1485] Step 2:

[1486] Information Extraction

[1487] The server receives user input and runs a Python script to query a MySQL database, extracting information related to specific keywords or phrases.

[1488] Input: User's inquiry

[1489] Data manipulation: A Python script queries the database and extracts relevant information (e.g., "SELECT date FROM meetings WHERE id = (SELECT MAX(id) FROM meetings)").

[1490] Output: Extracted information (e.g., next meeting date and time)

[1491] Step 3:

[1492] Emotion Engine Operation

[1493] The server sends the user's input text to the emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotion. The server receives the analysis results and recognizes the user's emotion.

[1494] Input: User-entered text

[1495] Data calculations: The sentiment engine analyzes the text and identifies sentiment (e.g., "interest" or "concern").

[1496] Output: User sentiment analysis results

[1497] Step 4:

[1498] Presentation of information

[1499] The server formats the extracted information in text format and responds to the user, providing appropriate information according to the user's emotions.

[1500] Input: Extracted information and sentiment analysis results

[1501] Data processing: Format the extracted information into text and generate an appropriate message for the user (e.g., "The next meeting will be held on October 15, 2023 at 2:00 PM").

[1502] Output: Messages sent to the user's terminal

[1503] Step 5:

[1504] User Visibility

[1505] The user's device receives the message sent from the server and displays it on the screen, allowing the user to check the necessary information.

[1506] Input: Message sent from the server

[1507] Output: Information displayed on the user's device (e.g., "The next meeting will be held on October 15, 2023 at 2:00 PM.")

[1508] In this way, users can quickly and accurately obtain the information they need. In addition, by using an emotion engine, it is possible to provide appropriate information according to the user's emotions.

[1509] (Application example 3)

[1510] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1511] Conventional meeting information management systems made it difficult for users to quickly obtain specific information, and were unable to provide integrated information across multiple meetings. Furthermore, they did not take user emotions into consideration, making it difficult to provide information tailored to user needs. Furthermore, factory robots could not optimize their work based on past meeting information, making it difficult to achieve efficient work.

[1512] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1513] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to the user, means for recognizing the user's emotions and extracting information based on those emotions, means for presenting the extracted information in text format, and means installed in the factory robot for optimizing work based on past conference information. This allows users to quickly obtain the information they need, enables information to be provided based on emotions, and enables factory robots to work more efficiently.

[1514] "Meeting information" refers to video and audio data that records the contents of the meeting.

[1515] The term "means for extracting information" refers to a method or device for extracting information related to a specific content from the conference information.

[1516] "Means for presenting to the user" refers to a method or device for showing the extracted information to the user.

[1517] "Means for recognizing emotions" refers to a method or device for analyzing a user's emotions and understanding those emotions.

[1518] "Means for presenting in text form" refers to a method or device for displaying extracted information to a user as text information.

[1519] A "factory robot" refers to an automated mechanical device designed to perform tasks in a factory.

[1520] "Means for optimizing work" refers to methods and devices that enable factory robots to perform work efficiently based on past meeting information.

[1521] In order to implement the present invention, the following hardware and software are used.

[1522] Hardware

[1523] Factory Robots

[1524] Audio input device

[1525] display

[1526] software

[1527] EmotionEngine (emotion recognition engine)

[1528] MeetingDataExtractor (meeting information extraction engine)

[1529] System configuration

[1530] The server receives multiple meeting information (video and audio recordings) as input and has a means to extract information related to specific content. When a user queries for a specific keyword or phrase, the server uses the EmotionEngine to recognize the user's emotion and extracts information based on that emotion. The extracted information is formatted in text format using the MeetingDataExtractor and displayed on the screen.

[1531] Processing flow

[1532] 1. The user queries the robot with a specific keyword or phrase through a voice input device.

[1533] 2. The server uses EmotionEngine to analyze the user's emotions.

[1534] 3. The server uses MeetingDataExtractor to extract the meeting information relevant to the query.

[1535] 4. The extracted information is formatted in text format and displayed on the screen.

[1536] 5. Factory robots will optimize their work based on past meeting information.

[1537] Specific examples

[1538] When a factory worker asks the robot about the "optimal procedure," the robot extracts information about the optimal procedure that was discussed in past meetings and presents it to the worker. For example, the following prompt sentence can be input into the generative AI model:

[1539] Prompt Sentence Examples

[1540] "Please give me some information on the best procedure."

[1541] Based on this prompt, the server extracts relevant procedures and important points from past meeting information and provides them to the worker, allowing the worker to proceed with their work efficiently.

[1542] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1543] Step 1:

[1544] The user queries the robot with specific keywords or phrases through a voice input device.

[1545] Input: User voice input (e.g., "Please give me some information about the best course of action.")

[1546] Output: Audio data

[1547] Specific operation: The user speaks a question into the voice input device, which converts this speech into digital data and sends it to the server.

[1548] Step 2:

[1549] The server uses the Emotion Engine to analyze the user's emotions.

[1550] Input: Audio data

[1551] Output: User's emotion data (e.g., frustrated)

[1552] How it works: The server inputs voice data into the Emotion Engine, which analyzes the user's emotions based on their tone of voice and choice of words. As a result of the analysis, the user's emotional data is generated.

[1553] Step 3:

[1554] The server uses the MeetingDataExtractor to extract the meeting information relevant to the query.

[1555] Input: Keywords and phrases, user sentiment data

[1556] Output: Extracted meeting information (e.g., text data of discussions on optimal procedures)

[1557] Specific operation: Based on the user's inquiry and emotion data, the server uses MeetingDataExtractor to search for related meeting information and extract the relevant information.

[1558] Step 4:

[1559] The extracted information is formatted in text format and displayed on the screen.

[1560] Input: Extracted meeting information

[1561] Output: Textual information (e.g., summary of discussion on optimal procedures)

[1562] Specific operation: The server converts the extracted meeting information into text format and generates data to be displayed on the display. This information is then displayed on the display so that the user can check it.

[1563] Step 5:

[1564] Factory robots optimize their work based on past meeting information.

[1565] Input: Text format information

[1566] Output: Optimized workflow

[1567] How it works: Based on the information displayed on the screen, the factory robot adjusts its work procedures and executes the work in the most optimal way, thereby improving work efficiency.

[1568] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1569] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1570] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1571] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1572] [Third embodiment]

[1573] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1574] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1575] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1576] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1577] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1578] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1579] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1580] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1581] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1582] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1583] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1584] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1585] "Example 1"

[1586] The system of the present invention has a means for receiving multiple pieces of conference information (video and audio recordings) as input. Specifically, a user inputs recording data of a conference held using a conference tool (e.g., Zoom) into the system.

[1587] "Example 2"

[1588] The system of the present invention has a means for extracting information related to specific content. Specifically, when a user makes a query such as "What did that person say?", the system extracts related content from the input conference information. This extraction is performed using voice recognition technology and natural language processing technology.

[1589] "Example 3"

[1590] The system of the present invention has a means for presenting the extracted information to the user. Specifically, the system presents the extracted information to the user in text format. The system can also present the date, time, participants, and other information of related meetings.

[1591] "Example 4"

[1592] The system of the present invention has a means for extracting information based on a user's inquiry. Specifically, when a user inquires about a specific keyword or phrase, the system extracts information related to that keyword or phrase from the conference information.

[1593] "Example 5"

[1594] The system of the present invention has a means for extracting information across multiple conference records. Specifically, when a user inquires about discussions that took place in multiple conferences, the system extracts related information from the multiple conference records. This allows the user to obtain information across multiple conferences at once, rather than just one conference.

[1595] The processing flow of each embodiment will be described below.

[1596] "Example 1"

[1597] Step 1: Users input recording data of meetings they have held using a conferencing tool (e.g., Zoom) into the system.

[1598] "Example 2"

[1599] Step 1: The user asks a question such as, "What did that person say?"

[1600] Step 2: The system extracts relevant content from the input meeting information using voice recognition and natural language processing technologies.

[1601] "Example 3"

[1602] Step 1: The system presents the extracted information to the user in text format.

[1603] Step 2: Provide relevant meeting dates, times, participants, and other information.

[1604] "Example 4"

[1605] Step 1: User queries for a specific keyword or phrase.

[1606] Step 2: The system extracts information related to the keyword or phrase from the meeting information.

[1607] "Example 5"

[1608] Step 1: A user inquires about discussions that took place across multiple meetings.

[1609] Step 2: The system extracts relevant information from multiple conferences, allowing users to obtain information across multiple conferences at once, rather than just a single conference.

[1610] Example 1

[1611] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1612] Conventional meeting information management systems have had difficulty efficiently extracting necessary information from recorded meeting data and generating summaries. Furthermore, they lacked a means for users to quickly grasp the content of a meeting. This resulted in the problem of requiring a great deal of time and effort when reviewing the content of a meeting later.

[1613] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1614] In this invention, the server includes a means for receiving multiple pieces of conference information (video and audio recordings) as input and converting the audio data into text, a means for inputting a prompt sentence into a generative AI model to generate a summary, and a means for presenting the generated summary to the user. This allows for efficient generation of summaries from recorded conference data, enabling users to quickly understand the content of the conference.

[1615] "Meeting information" refers to video and audio data that records the contents of the meeting.

[1616] "Means for converting voice data to text" refers to technology or devices for analyzing voice data and generating corresponding text data.

[1617] A "generative AI model" refers to an artificial intelligence model that performs natural language processing based on input data to generate summaries and other text.

[1618] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[1619] "Means for generating a summary" refers to technology or equipment for creating a summary from input text data using a generative AI model.

[1620] "User presentation means" refers to the techniques and devices used to display or provide the generated summary to the user.

[1621] This invention begins when a user uploads a video file of a conference recorded with a conference tool to the system. The user logs in to the system's web portal, selects the recorded data, and clicks the upload button. The server receives the conference recording data uploaded by the user and temporarily stores it.

[1622] The server then converts the audio portion of the recording into text using the Google Cloud Speech-to-Text API. Specifically, the server extracts the audio track from the recording and sends it to the API, which parses the audio data and returns the corresponding text data.

[1623] The server then inputs a prompt to the generative AI model (e.g., OpenAI's GPT-4). The prompt includes the converted text data and instructions for generating a summary. For example, the prompt might read, "The following text is the content of a one-hour meeting. Please write a summary of this meeting in 500 characters or less."

[1624] The generative AI model generates a summary of the meeting based on this prompt. The server receives the summary and provides it to the user, who can review it through the system's interface. The server then associates the summary with the user's account and saves it for later access.

[1625] As a concrete example, consider the case where a user uploads a recording of a one-hour Zoom meeting to the system. The server first converts the hour's worth of audio data into text using the Google Cloud Speech-to-Text API. Then, it inputs the following prompt to the generative AI model (GPT-4):

[1626] "The text below is from a one-hour meeting. Please write a summary of this meeting in 500 words or less."

[1627] The generative AI model generates a summary of the meeting based on this prompt, and the server provides the generated summary to the user, who can then view it through the system interface.

[1628] The above is a specific embodiment for carrying out the present invention. This system makes it possible to efficiently generate summaries from recorded data of a conference, enabling users to quickly grasp the contents of the conference.

[1629] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1630] Step 1:

[1631] A user uploads a meeting recording.

[1632] A user logs in to the system's web portal, selects the recording data, and clicks the upload button. The input is the meeting recording data selected by the user (e.g., meeting_recording.mp4), and the output is the recording data sent to the server.

[1633] Step 2:

[1634] The server receives the recording data.

[1635] The server receives the conference recording data uploaded by the user and temporarily stores it. The input is the recording data sent by the user, and the output is the recording data saved in the server's storage.

[1636] Step 3:

[1637] The server converts the audio data into text.

[1638] The server converts the audio portion of the recording into text using the Google Cloud Speech-to-Text API. Specifically, the server extracts the audio track from the recording and sends it to the API. The input is the audio track of the recording, and the output is the text data returned by the API.

[1639] Step 4:

[1640] The server inputs a prompt sentence into the generative AI model.

[1641] The server inputs a prompt to the generative AI model (e.g., OpenAI's GPT-4). The prompt includes the converted text data and instructions to generate a summary. The input is the text data and the prompt, and the output is the summary returned by the generative AI model. For example, the prompt might be, "The following text is the content of a one-hour meeting. Please write a summary of this meeting in 500 characters or less."

[1642] Step 5:

[1643] The server provides the generated summary to the user.

[1644] The server receives the summary returned by the generative AI model and provides it to the user. The user can review the summary through the system's interface. The input is the summary returned by the generative AI model, and the output is the summary provided to the user. The server associates the summary with the user's account and stores it for later access by the user.

[1645] (Application example 1)

[1646] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1647] Conventional meeting information management systems simply store recorded meeting data, making it difficult to quickly extract necessary information later. Furthermore, they lack a function to summarize and report the contents of meetings, which means that users have to spend a lot of time reviewing the entire contents of a meeting. This can lead to a decrease in efficiency, especially in situations where quick decision-making is required, such as in factories.

[1648] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1649] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input, extracting information related to specific content, presenting the extracted information to the user, summarizing the extracted information, and reporting the summarized information to the user. This makes it possible to quickly extract necessary information from recorded conference data and summarize and report it.

[1650] "Multiple conference information" refers to video and audio data recorded in multiple conferences.

[1651] "Means of receiving input" refers to the interfaces and processes used to input meeting information into the system.

[1652] "Means for extracting information related to specific content" refers to algorithms or techniques for identifying and extracting information related to specific topics or keywords from meeting information.

[1653] "Means for presenting extracted information to a user" refers to an interface or method for providing extracted information to a user visually or audibly.

[1654] "Means for summarizing extracted information" refers to algorithms or techniques for concisely summarizing extracted information.

[1655] "Means for reporting summarized information to the user" refers to the interface or method for notifying the user of the summarized information.

[1656] A system for implementing this invention receives multiple pieces of conference information (video and audio recordings) as input, extracts information related to specific content, summarizes the extracted information, and reports it to the user. A specific embodiment of this system is described below.

[1657] Hardware and software used

[1658] Hardware:

[1659] Factory robots (e.g. KUKA, Fanuc, etc.)

[1660] Audio Capture Device

[1661] Computer (may be built into the robot)

[1662] software:

[1663] Python

[1664] moviepy (video processing library)

[1665] speech_recognition (speech recognition library)

[1666] transformers (AI models for generating Hugging Face)

[1667] Data processing and calculation

[1668] 1. Extract audio from video:

[1669] The server uses moviepy to extract audio from the conference video and generate a temporary audio file, which is used for subsequent speech recognition processing.

[1670] 2. Speech Recognition:

[1671] The server uses the speech_recognition library to convert the extracted speech into text, which is necessary to handle the contents of the meeting as text information.

[1672] 3. Text Summary:

[1673] The server uses Hugging Face's transformers library to summarize the text using a generative AI model (BART), and this summarized text is reported back to the user.

[1674] 4. Information Presentation and Reporting:

[1675] The server provides an interface for presenting the summarized information to the user, specifically by sending a notification to the user's terminal and displaying the summarized meeting content.

[1676] Specific examples

[1677] Factory managers input the video data of weekly production meetings into a robot. The robot analyzes the video data, summarizes important topics and decisions, and reports them to managers. This saves managers the time of having to review the entire meeting content, allowing them to quickly grasp only the important information.

[1678] Prompt Sentence Examples

[1679] An example of a prompt to input to a generative AI model is as follows:

[1680] "Please summarize the following text:"

[1681] Text extracted from meeting recordings

[1682] This prompt is used to ask the generative AI model to create a summary.

[1683] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1684] Step 1:

[1685] The server receives a conference video file from the user. The input is the conference video file, and the output is the saving of the video file. Specifically, the user uploads the video file recorded using the conference tool to the server.

[1686] Step 2:

[1687] The server uses moviepy to extract audio from conference videos. The input is the conference video file, and the output is an audio file. Specifically, the server reads the video file, separates the audio track, and generates a temporary audio file.

[1688] Step 3:

[1689] The server uses the speech_recognition library to convert the extracted audio into text. The input is an audio file, and the output is text data. Specifically, the server passes the audio file to a speech recognition engine, which converts the audio data into text information.

[1690] Step 4:

[1691] The server uses Hugging Face's transformers library to summarize text using a generative AI model (BART). The input is text data, and the output is summarized text. Specifically, the server inputs text data into the generative AI model and obtains the summarized text.

[1692] Step 5:

[1693] The server provides an interface for reporting the summarized information to the user. The input is the summary text, and the output is a notification to the user terminal. In concrete terms, the server sends the summary text to the user terminal and displays the notification.

[1694] Example 2

[1695] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1696] Conventional meeting information management systems make it difficult for users to quickly and accurately extract information related to specific content. Furthermore, the process of converting audio data into text data, analyzing that text data, and extracting relevant information is complicated, making it difficult to provide appropriate information in response to user inquiries. This makes it difficult to efficiently grasp the content of meetings, hindering work efficiency.

[1697] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1698] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for converting the audio data into text data, means for analyzing the text data and extracting related information based on a user's inquiry, and means for presenting the extracted information to the user, thereby enabling the user to quickly and accurately extract information related to specific content.

[1699] "Meeting information" refers to video and audio data that records the contents of the meeting.

[1700] "Content-specific information" refers to information related to a particular topic or question that interests a user.

[1701] "Audio data" refers to digital audio files that record statements and discussions made during a meeting.

[1702] "Text data" refers to data obtained by converting voice data into character information.

[1703] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into corresponding text data.

[1704] "Natural language processing technology" refers to the technology of analyzing text data, understanding meaning and context, and extracting information.

[1705] "User Query" means a question or request entered by a User into the System requesting specific information.

[1706] "Means for extracting related information" refers to methods and technologies for searching and extracting related information from text data based on a user's inquiry.

[1707] "Means for presenting information" refers to the methods and techniques used to display extracted information to the user.

[1708] MODE FOR CARRYING OUT THE INVENTION

[1709] This invention relates to a system that receives meeting information (video and audio recordings) as input from a user, extracts information related to specific content, and presents it to the user. A specific embodiment of this system is described below.

[1710] System configuration

[1711] Hardware and software configuration:

[1712] 1. Server:

[1713] The server receives the voice data and converts it into text data using voice recognition technology.

[1714] The server analyzes the converted text data using natural language processing technology and extracts relevant information based on the user's inquiry.

[1715] The server presents the extracted information to the user.

[1716] 2. Terminal:

[1717] A user uses a terminal to upload the conference information to the server.

[1718] The user checks the information provided by the server through the terminal.

[1719] Specific techniques used:

[1720] As speech recognition technology, we use a general speech recognition API (e.g., Google Cloud Speech-to-Text API).

[1721] As a natural language processing technique, we use generative AI models (e.g., OpenAI's GPT-4 model).

[1722] System Operation

[1723] Upload audio data:

[1724] Users upload the audio data of the conference from their devices to the server. The audio data is typically provided in MP3 or WAV format. Users select the file through the system interface and click the upload button.

[1725] Audio data conversion:

[1726] The server receives the uploaded voice data and converts it into text data using voice recognition technology, extracting the spoken content from the voice data as text.

[1727] Analyzing text data:

[1728] The server then analyzes the converted text data using natural language processing technology, specifically using a generative AI model to extract information relevant to the user's inquiry.

[1729] Extracting relevant information:

[1730] The server searches for and extracts relevant comments from text data based on the user's inquiry. For example, in response to a query such as "What did that person say?", it extracts comments from a specific person.

[1731] Providing results:

[1732] The server returns the extracted relevant information to the user, who can check the results through their terminal.

[1733] Specific examples

[1734] If a user uploads audio from a meeting and asks, "What did they say about the progress of the project?", the system will proceed as follows:

[1735] 1. The user opens the browser on the terminal and accesses the system's web page. The user clicks the "Choose File" button and selects an audio file (e.g., "meeting_audio.mp3") from the local disk. Then, the user clicks the "Upload" button.

[1736] 2. When the server receives the voice data, it sends it to the speech recognition API. The API analyzes the voice data and returns the corresponding text data (e.g., "The project is 50% complete."). The server stores this text data.

[1737] 3. The server receives the user's query (e.g., "What did he say about the progress of the project?"). The server inputs the text data and the query into the generative AI model and extracts relevant utterances. For example, the server obtains a result such as, "Regarding the progress of the project, he said: 'Currently, progress is at 50%.'"

[1738] 4. The server sends the extracted information to the user's device. The user can check the results through the system interface. For example, a message like "The project progress was reported as follows: 'Currently, progress is 50%.'" will be displayed on the browser.

[1739] Example prompt sentence:

[1740] "I've uploaded the audio of the meeting. Can you tell me what they said about the project's progress?"

[1741] In this way, the server realizes a system that provides appropriate information in response to user inquiries.

[1742] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1743] Program processing flow

[1744] Step 1: Upload your audio data

[1745] Specific behavior:

[1746] The user opens the browser on the terminal and accesses the system's web page. The user clicks the "Choose File" button and selects an audio file (e.g., "meeting_audio.mp3") from the local disk. Then, the user clicks the "Upload" button.

[1747] input:

[1748] An audio file selected by the user (e.g., "meeting_audio.mp3")

[1749] output:

[1750] Audio files uploaded to the server

[1751] Step 2: Receiving and storing audio data

[1752] Specific behavior:

[1753] The server receives the voice data uploaded by the user and stores it in a specified directory.

[1754] input:

[1755] User-uploaded audio files

[1756] output:

[1757] Audio files stored on the server

[1758] Step 3: Convert audio data to text

[1759] Specific behavior:

[1760] The server sends the stored voice data to a voice recognition API (e.g., a general voice recognition API), which analyzes the voice data and returns the corresponding text data.

[1761] input:

[1762] Audio files stored on the server

[1763] Data processing:

[1764] Convert voice data into text using speech recognition API

[1765] output:

[1766] Text data (e.g., "Project progress is 50%.")

[1767] Step 4: Save the text data

[1768] Specific behavior:

[1769] The server receives the text data returned from the speech recognition API and stores it in a database or file system.

[1770] input:

[1771] Text data returned from the speech recognition API

[1772] output:

[1773] Text data stored on the server

[1774] Step 5: Receiving user inquiries

[1775] Specific behavior:

[1776] A user enters a query through the system interface, for example, "What did they say about the progress of the project?"

[1777] input:

[1778] What the user is asking (e.g., "What did they say about the project status?")

[1779] output:

[1780] User inquiry sent to the server

[1781] Step 6: Analyze text data and extract relevant information

[1782] Specific behavior:

[1783] The server inputs the stored text data and the user's query into a generative AI model (e.g., OpenAI's GPT-4 model) to extract relevant information.

[1784] input:

[1785] Text data stored on the server, user inquiries

[1786] Data Calculation:

[1787] Analyze text data using generative AI models to extract relevant information

[1788] output:

[1789] Extracted relevant information (e.g., "Regarding the project progress, they said: 'Currently, it's 50% complete.'")

[1790] Step 7: Present the extracted information

[1791] Specific behavior:

[1792] The server sends the extracted relevant information to the user's terminal, and the user can check the results through the system interface.

[1793] input:

[1794] Extracted related information

[1795] output:

[1796] Relevant information displayed on the user's device (e.g., "Regarding the project's progress, they said: 'Currently, it's 50% complete.'")

[1797] In this way, the server realizes a system that provides appropriate information in response to user inquiries.

[1798] (Application example 2)

[1799] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1800] With conventional meeting information management systems, it was difficult to quickly and accurately extract specific content when searching for meeting details later. Furthermore, few systems utilized voice recognition or natural language processing technology, making it difficult to provide appropriate answers to user inquiries. Furthermore, they lacked the functionality to extract information across multiple meeting information sources, resulting in low user convenience.

[1801] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to the user, means for converting speech to text using speech recognition technology, and means for generating answers to user questions using natural language processing technology. This allows the user to quickly and accurately search for specific content and extract information across multiple pieces of conference information, improving convenience.

[1802] "Meeting information" refers to video and audio data recorded during a meeting.

[1803] "Information related to specific content" is a part of the conference information related to the content inquired by the user.

[1804] The "means for extracting information" refers to a technique or method for extracting information related to specific content from the conference information.

[1805] "Means for presenting to the user" refers to the techniques or methods for displaying the extracted information to the user.

[1806] "Speech recognition technology" is a technology that converts voice data into text data.

[1807] "Natural language processing technology" is a technology that analyzes text data and generates appropriate answers to user questions.

[1808] A "user question" is the content of an inquiry that a user makes to the system.

[1809] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[1810] The term "means for extracting information across multiple pieces of conference information" refers to a technique or method for extracting information related to specific content from multiple pieces of conference information.

[1811] A system for implementing this invention is configured as follows: The server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to a user, means for converting speech into text using speech recognition technology, and means for generating answers to user questions using natural language processing technology.

[1812] First, the server uses speech recognition technology to convert the audio data of the conference information into text data. For example, the "speech_recognition" library can be used as the speech recognition technology. The server reads the audio file and converts the audio data into text.

[1813] The server then uses natural language processing (NLP) techniques, such as the "pipeline" technique from the "transformers" library, to generate an answer to the user's question. Specifically, the server uses the "deepset / roberta-base-squad2" model to generate an appropriate answer to the user's question.

[1814] When a user wants to search for a specific topic, the server receives the user's question and uses speech recognition and natural language processing technologies to extract relevant information from the meeting information and present it to the user, allowing the user to search for the specific topic quickly and accurately.

[1815] As a concrete example, consider the case where a search is made for "discussions made about the installation of a new production line" during a meeting in a factory. When a user asks, "What was said about the installation of a new production line?", the server extracts relevant content from the meeting information and replies, "The installation of the new production line is scheduled to begin next month, with a budget of 5 million yen."

[1816] Example prompt sentence:

[1817] User: What did they say about setting up a new production line?

[1818] System: Installation of the new production line is scheduled to begin next month, with a budget of 5 million yen.

[1819] In this way, the server can combine speech recognition and natural language processing technologies to generate and present appropriate answers to user questions, enabling users to search for information across multiple conference information sources, improving convenience.

[1820] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1821] Step 1:

[1822] The server receives multiple meeting information (video and audio recordings) as input. Specifically, the server receives audio and video files uploaded by users. This input data is used in subsequent processing steps.

[1823] Step 2:

[1824] The server uses speech recognition technology to convert the received voice data into text data. Specifically, it uses the "speech_recognition" library to read the audio file and convert the voice data into text. The input of this step is voice data, and the output is text data.

[1825] Step 3:

[1826] The server stores the converted text data, which is used in a subsequent natural language processing step, whose input is the text data and whose output is the stored text data.

[1827] Step 4:

[1828] When a user wants to search for something specific, they enter a question. Specifically, the user asks the system, "What did they say about installing a new production line?" The input for this step is the user's question, and the output is the question text.

[1829] Step 5:

[1830] The server uses natural language processing techniques to generate answers to user questions. Specifically, it uses the "pipeline" from the "transformers" library and the "deepset / roberta-base-squad2" model to generate appropriate answers to questions. The input for this step is the question text and stored text data, and the output is the generated answer.

[1831] Step 6:

[1832] The server presents the generated answer to the user. Specifically, it displays the answer on the user's screen. The input of this step is the generated answer, and the output is the answer presented to the user.

[1833] Step 7:

[1834] The user can review the presented answers and ask further questions if necessary. The input of this step is the answers presented to the user, and the output is the user's review.

[1835] In this way, the server can combine speech recognition and natural language processing technologies to generate and present appropriate answers to user questions, enabling users to search for information across multiple conference information sources, improving convenience.

[1836] Example 3

[1837] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1838] Conventional meeting information management systems make it difficult for users to quickly and accurately obtain specific information. They also lack the functionality to extract related information across multiple meeting records, preventing users from obtaining all the information they need at once. Furthermore, they lacked the means to present the extracted information in an easy-to-understand text format, resulting in low user convenience.

[1839] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1840] In this invention, the server includes means for receiving multiple pieces of conference information as input and extracting information related to specific content, means for extracting information based on user inquiries, means for analyzing the extracted information using a natural language processing engine, means for converting the analyzed information into text format using a generative AI model, and means for presenting the converted text to the user. This allows the user to quickly and accurately obtain specific information and to simultaneously obtain related information across multiple pieces of conference information. Furthermore, presenting the extracted information in an easy-to-understand text format improves user convenience.

[1841] "Meeting information" refers to recorded data such as audio, video, and text related to the meeting.

[1842] "Means for extracting information" refers to techniques and methods for extracting information related to specific content from the meeting information.

[1843] "User Enquiry" refers to keywords or phrases that a user enters into the system to obtain specific information.

[1844] A "natural language processing engine" refers to software that analyzes user inquiries and extracted information and understands their meaning.

[1845] "Generative AI model" refers to an artificial intelligence model that converts analyzed information into natural language text format.

[1846] "Means for converting to text format" refers to the techniques and methods for converting extracted information into a text format that is easy for users to understand.

[1847] "Means for presenting to the user" refers to the technology or method for displaying the converted text on the user's device.

[1848] "Means for extracting information across multiple conference records" refers to techniques and methods for extracting related information from multiple conference records at once.

[1849] "Server" means a computer system that receives user queries and extracts, analyzes, converts, and presents information.

[1850] "Terminal" refers to a device through which a user can enter queries and receive results.

[1851] The present invention relates to a system for extracting conference information based on an inquiry from a user and presenting it in text format. A specific embodiment of this system will be described below.

[1852] First, this system uses the following hardware and software:

[1853] Hardware: Servers, terminals

[1854] Software: Database Management System (DBMS), Natural Language Processing (NLP) engine, Generative AI model

[1855] The server receives inquiries from users. When the user inputs specific keywords or phrases, the server sends the information to an NLP engine. The NLP engine extracts relevant meeting information from a database based on the input keywords or phrases. The extracted information is converted into text using a generative AI model. The server then sends the generated text to the user's device.

[1856] As a concrete example, the following processing occurs when a user asks, "Who are the participants in the next meeting?"

[1857] 1. The user types into the device interface, "Who's in my next meeting?"

[1858] 2. The device sends this query to the server.

[1859] 3. The server receives the query and sends it to the NLP engine.

[1860] 4. The NLP engine analyzes the keywords "next meeting" and "participants" and extracts relevant meeting information from the database.

[1861] 5. The server sends the extracted information to the generative AI model.

[1862] 6. The generative AI model generates the text, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1863] 7. The server sends the generated text to the terminal.

[1864] 8. The device displays to the user, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1865] In this way, the user can easily obtain the information he or she needs.

[1866] An example of a prompt sentence might be:

[1867] Who will be at your next meeting?

[1868] After entering this prompt, the system will present the user with a list of participants for the next meeting in text format. For example, the result might look like this: "The participants for the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1869] This system allows users to quickly and accurately obtain specific information, and to simultaneously obtain related information across multiple conference information. Furthermore, by presenting the extracted information in an easy-to-understand text format, user convenience is improved. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[1870] Step 1:

[1871] The user enters a query.

[1872] The user uses the device's interface to enter a query containing specific keywords or phrases, for example, "Who are the participants in the next meeting?" The entered data is temporarily stored in the device's memory.

[1873] Step 2:

[1874] The terminal sends a query to the server.

[1875] The terminal sends the query entered by the user to the server using a communication protocol such as an HTTP request. The input data is transferred to the server via the network.

[1876] Step 3:

[1877] The server receives the query and sends it to the NLP engine.

[1878] The server analyzes queries received from the device and sends them to the NLP engine. The server analyzes the received data and converts it into a format that the NLP engine can understand. The converted data is then passed to the NLP engine.

[1879] Step 4:

[1880] The NLP engine analyzes keywords and extracts information from the database.

[1881] The NLP engine analyzes the data received from the server and extracts specific keywords and phrases. For example, it searches the database based on keywords like "next meeting" or "participants." It then extracts relevant meeting information from the database and returns that information to the server.

[1882] Step 5:

[1883] The server sends the extracted information to the generative AI model.

[1884] The server sends the extracted information received from the NLP engine to the generative AI model. The server converts the extracted data into a format that the generative AI model can understand and passes it to the generative AI model.

[1885] Step 6:

[1886] The generative AI model converts it into text format.

[1887] The generative AI model converts the information received from the server into a text format that is easy for the user to understand. For example, it generates text such as, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato." The generated text is then returned to the server.

[1888] Step 7:

[1889] The server sends the generated text to the terminal.

[1890] The server sends the text received from the generative AI model to the device, again using a communication protocol such as an HTTP request. The generated text is then transferred to the device via the network.

[1891] Step 8:

[1892] The terminal displays the results to the user.

[1893] The terminal displays the text received from the server to the user. The user can check the results on the terminal screen. For example, the result may read, "The participants in the next meeting are Taro Tanaka, Ichiro Suzuki, and Hanako Sato."

[1894] (Application example 3)

[1895] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1896] There is a need to efficiently manage information on multiple meetings and discussions within a factory and quickly extract and present information related to specific keywords. However, conventional systems have had issues with cross-sectional searches of multiple meeting information, making it difficult to quickly obtain the necessary information. Furthermore, they lacked the functionality to present the extracted information along with the date, time, and participant information of related meetings, making it difficult for users to comprehensively grasp the information they need.

[1897] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1898] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input and extracting information related to specific content, means for presenting the extracted information to a user, means for presenting the extracted information along with the date and time of related conferences and attendee information, means for extracting information across multiple pieces of conference information, means for a user to input specific keywords or phrases, and means for searching for information based on the keywords or phrases input by the user. This allows a user to efficiently manage information on multiple conferences and meetings within a factory and quickly obtain information related to specific keywords.

[1899] "Meeting information" refers to video and audio data recorded during meetings and discussions.

[1900] The "means for extracting information" refers to a method or device for extracting information related to specific content from input conference information.

[1901] "Means for presenting to the user" refers to a method or device for visually or audibly presenting the extracted information to the user.

[1902] The "means for presenting the date and time and participant information" refers to a method or device for providing the user with the date and time and participant information of the related conference together with the extracted information.

[1903] The "means for extracting information across multiple conferences" refers to a method or device for extracting related information collectively from multiple conference information.

[1904] A "means for entering keywords or phrases" is a method or device that allows a user to enter specific keywords or phrases into the system.

[1905] "Means for searching information" refers to a method or device for retrieving relevant information from conference information based on keywords or phrases entered by the user.

[1906] The system for implementing this invention aims to efficiently manage information on multiple meetings and conferences within a factory and quickly extract and present information related to specific keywords. A specific embodiment of this system will be described below.

[1907] System configuration

[1908] The system consists of the following major components:

[1909] 1. Database server: Manages the database for storing conference information (video and audio recordings).

[1910] 2. Search server: Searches conference information and extracts relevant information based on keywords and phrases provided by the user.

[1911] 3. User Interface: Provides an interface for users to enter keywords or phrases and view search results.

[1912] Hardware and software used

[1913] Hardware: Factory robots, servers

[1914] Software: SQLite (database management system), Python (programming language)

[1915] Data processing and calculation flow

[1916] 1. Database connection: The database server uses SQLite to store conference information, including video and audio recordings, conference dates and times, and participant information.

[1917] 2. Information extraction: The search server extracts relevant information from the conference information based on the keywords or phrases entered by the user, specifically by using SQL queries to search for detailed conference information.

[1918] 3. Result presentation: The extracted information is presented to the user through a user interface, including the meeting date, time, participants, and details.

[1919] Specific examples

[1920] For example, if a factory manager wants to search for past meeting information using the keyword "quality control," he or she would perform the following operations.

[1921] 1. Enter the keyword "quality control" into the user interface.

[1922] 2. The search server connects to the database server and searches for relevant conference information.

[1923] 3. The extracted information, along with the meeting date, time, and participant information, is displayed in the user interface.

[1924] Prompt Sentence Examples

[1925] "Find information about quality control discussed at past meetings."

[1926] By inputting this prompt into a generative AI model, relevant meeting information can be quickly retrieved.

[1927] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1928] Step 1:

[1929] A user uses a terminal to enter a particular keyword or phrase into a user interface.

[1930] Input: Keyword or phrase (e.g., "quality control")

[1931] Output: The keyword or phrase you entered

[1932] Step 2:

[1933] The terminal transmits the entered keywords or phrases to a search server.

[1934] Input: Keywords or phrases entered by the user

[1935] Output: Keywords or phrases sent to the search server

[1936] Step 3:

[1937] The search server connects to a database server and searches for relevant conference information based on the received keywords or phrases.

[1938] Input: Submitted keyword or phrase

[1939] Output: Search results using SQL queries (related conference information)

[1940] Step 4:

[1941] The database server searches the conference information and extracts relevant information based on the SQL query from the search server.

[1942] Input: SQL query

[1943] Output: Extracted meeting information (meeting date, time, participants, details)

[1944] Step 5:

[1945] The search server processes the extracted information received from the database server and converts it into a format for presentation to the user.

[1946] Input: Extracted meeting information

[1947] Output: Information transformed into a format for presentation to the user

[1948] Step 6:

[1949] The search server transmits the converted information to the terminal.

[1950] Input: Formatted information

[1951] Output: Information sent to the terminal

[1952] Step 7:

[1953] The terminal displays the received information on a user interface and presents it to the user.

[1954] Input: Received information

[1955] Output: Information displayed in the user interface (meeting date, time, participants, details)

[1956] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1957] "Example 1"

[1958] In one embodiment of the present invention, a recording of a meeting held by a user using a conferencing tool (e.g., Zoom) is input to the system. The system includes an emotion engine that recognizes emotions from the user's voice.

[1959] "Example 2"

[1960] When a user asks a question such as "What did that person say?", the system extracts relevant content from the input meeting information. This extraction is done using speech recognition and natural language processing technologies. Furthermore, the emotion engine recognizes emotions from the user's voice and extracts information based on those emotions.

[1961] "Example 3"

[1962] The system presents the extracted information to the user in text format, along with relevant meeting dates, participants, and other information. The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[1963] "Example 4"

[1964] When users query for specific keywords or phrases, the system extracts information related to those keywords or phrases from the meeting information.The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[1965] "Example 5"

[1966] When a user inquires about discussions that took place in multiple meetings, the system extracts relevant information from multiple meeting records. This allows users to obtain information across multiple meetings at once, rather than just one. The emotion engine also recognizes the user's emotions and extracts information based on those emotions.

[1967] The processing flow of each embodiment will be described below.

[1968] "Example 1"

[1969] Step 1: Users input recording data of meetings they have held using a conferencing tool (e.g., Zoom) into the system.

[1970] Step 2: The system activates an emotion engine that recognizes emotions from the user's voice.

[1971] Step 1: The user asks a question such as, "What did that person say?"

[1972] Step 2: The system extracts relevant content from the input meeting information using voice recognition and natural language processing technologies.

[1973] Step 3: The emotion engine recognizes emotions from the user's voice and extracts information based on those emotions.

[1974] "Example 3"

[1975] Step 1: The system presents the extracted information to the user in text format.

[1976] Step 2: Provide relevant meeting dates, times, participants, and other information.

[1977] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[1978] "Example 4"

[1979] Step 1: User queries for a specific keyword or phrase.

[1980] Step 2: The system extracts information related to the keyword or phrase from the meeting information.

[1981] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[1982] "Example 5"

[1983] Step 1: A user inquires about discussions that took place across multiple meetings.

[1984] Step 2: The system extracts relevant information from multiple conferences, allowing users to obtain information across multiple conferences at once, rather than just a single conference.

[1985] Step 3: The emotion engine recognizes the user's emotion and extracts information based on that emotion.

[1986] Example 1

[1987] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1988] Conventional meeting recording systems simply store the contents of the meeting, making it difficult to grasp the emotional state of the participants during the meeting. This makes it difficult to properly evaluate the effectiveness of the meeting or the reactions of the participants, and makes it difficult to obtain specific feedback to improve the quality of the meeting.

[1989] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1990] In this invention, the server includes means for receiving multiple pieces of conference information (video and audio recordings) as input, separating the video data from the audio data, means for analyzing the audio data and recognizing emotions, means for storing the recognized emotions in a database, and means for presenting the stored emotion analysis results to the user, thereby enabling automatic analysis and visualization of the emotional states of participants during a conference.

[1991] "Meeting information" refers to video and audio data recorded during a meeting.

[1992] "Video data" refers to data that includes video information from a conference.

[1993] "Audio data" refers to data containing audio information during a conference.

[1994] "Means for separating" refers to the technical means for extracting video data and audio data separately.

[1995] "Means for analyzing" refers to the technical means for analyzing the audio data and extracting specific information.

[1996] "Means for recognizing emotions" refers to technical means for identifying emotional states from audio data.

[1997] "Database" refers to a data storage system for storing analytical results and other information.

[1998] "Presentation Means" refers to the technical means by which information is displayed or presented to a user.

[1999] "User" refers to any individual or organization that uses the System.

[2000] "Query" refers to a request for information made by a user to a system.

[2001] "Means for extracting information across multiple conferences" refers to technical means for comparing and analyzing multiple conference information to extract common or related information.

[2002] MODE FOR CARRYING OUT THE INVENTION

[2003] This invention is a system that receives multiple pieces of conference information (video and audio recordings) as input, separates the video data from the audio data, analyzes the audio data to recognize emotions, and presents the results to the user.

[2004] Hardware and software used

[2005] The server uses the following hardware and software:

[2006] Hardware: powerful processor, sufficient memory, storage devices, and network interfaces

[2007] Software: FFmpeg (video processing library), Google Cloud Speech-to-Text API, IBM Watson sentiment analysis API, database management system (e.g., MySQL)

[2008] Data processing and calculation

[2009] The server receives the meeting recording data uploaded by the user and performs the following processes:

[2010] 1. Separating video and audio data: Use FFmpeg to separate video and audio data. The audio data is converted to WAV format.

[2011] 2. Analyze voice data: Use Google Cloud Speech-to-Text API and IBM Watson sentiment analysis API to analyze voice data and identify the user's emotional state.

[2012] 3. Save to database: The analysis results are saved to a database, including the meeting timestamp, emotion type, and emotion intensity.

[2013] 4. Presentation to the user: Based on the saved sentiment analysis results, feedback is provided to the user. The feedback is generated as a report visualizing changes in sentiment and displayed on the user's dashboard.

[2014] Specific examples

[2015] As a concrete example, consider a scenario in which a user uploads recording data of a meeting held using a conferencing tool to the system. When the user uploads the recording data of the meeting to the system, the server receives the data and inputs the voice data into the emotion engine. The emotion engine recognizes the user's emotion from the voice data and stores the result in a database. Finally, the user can receive a report visualizing the changes in emotion during the meeting.

[2016] Prompt Sentence Examples

[2017] Users upload recordings of meetings held using a conferencing tool to the system. The system analyzes the audio data and uses an emotion engine to recognize the user's emotions. The analysis results are stored in a database and a report is generated that visualizes changes in emotions.

[2018] In this way, it is possible to specifically explain how the server, terminal, and user work together to utilize the system.

[2019] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2020] Step 1:

[2021] A user uploads a meeting recording.

[2022] The user accesses the system's web interface, selects the conference recording data, and clicks the upload button. Once the upload is complete, the server receives the data.

[2023] Input: Meeting recording data (video and audio recording)

[2024] Output: Meeting recording data saved on the server

[2025] Step 2:

[2026] The server receives the recorded data and separates the video data from the audio data.

[2027] The server temporarily stores the received recording data and calls the FFmpeg library to separate the video data from the audio data. The audio data is saved as a WAV file.

[2028] Input: Meeting recording data stored on the server

[2029] Output: Separated video and audio data (WAV format)

[2030] Step 3:

[2031] The server inputs the voice data into the emotion engine.

[2032] The server sends the stored voice data to the emotion engine, which receives the voice data and starts analyzing it.

[2033] Input: Separated audio data (WAV format)

[2034] Output: Voice data input to the emotion engine

[2035] Step 4:

[2036] The emotion engine analyzes voice data and recognizes emotions.

[2037] The emotion engine analyzes the audio data and identifies the user's emotion at each timestamp, for example, recognizing that the user is feeling "joyful" five minutes into the meeting.

[2038] Input: Speech data input to the emotion engine

[2039] Output: Parsed emotion data (timestamp, emotion type, emotion intensity)

[2040] Step 5:

[2041] The server stores the analysis results in a database.

[2042] The server stores the analysis results received from the emotion engine in a database, including the meeting timestamp, emotion type, and emotion intensity.

[2043] Input: Parsed emotion data

[2044] Output: Sentiment analysis results stored in a database

[2045] Step 6:

[2046] The server provides feedback on the emotion analysis results to the user.

[2047] The server provides feedback to the user based on the emotion analysis results stored in the database. The feedback is generated as a report visualizing changes in emotion and displayed on the user's dashboard.

[2048] Input: Sentiment analysis results stored in the database

[2049] Output: Sentiment analysis results presented to the user (visual report)

[2050] (Application example 1)

[2051] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2052] With conventional conferencing systems, it was difficult to grasp the emotions of participants in real time during a meeting, and it was particularly difficult to detect abnormal emotional changes (e.g., anger, fear, stress) and take appropriate measures. Furthermore, it was not possible to take user emotions into consideration when extracting information related to specific content from recorded meeting data, making it impossible to improve the quality and safety of meetings.

[2053] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2054] In this invention, the server includes a means for receiving multiple pieces of conference information (video and audio recordings) as input, extracting information related to specific content, presenting the extracted information to the user, recognizing emotions from the user's voice, and issuing an alert if the recognized emotion is abnormal. This makes it possible to grasp the emotions of participants during a conference in real time, detect abnormal emotional changes, and take appropriate action. Furthermore, the user's emotions can be taken into consideration when extracting information related to specific content from the conference recording data, thereby improving the quality and safety of the conference.

[2055] "Meeting information" refers to video and audio data recorded during a meeting.

[2056] "Information related to a specific content" refers to information extracted from the conference information that is related to a specific topic or theme that is of interest to the user.

[2057] The term "extraction means" refers to a technical means for extracting information related to specific content from the conference information.

[2058] "Presentation means" refers to the technical means by which the extracted information is presented to the user visually or audibly.

[2059] "Means for recognizing emotions" refers to technical means for analyzing emotions from a user's voice data and identifying a specific emotional state.

[2060] "Means for issuing an alert" refers to a technical means for issuing a warning to a user or administrator when a recognized emotion is abnormal.

[2061] The system for implementing this invention receives multiple meeting information (video and audio recordings) as input, extracts information related to specific content, and presents it to the user. It also has the function of recognizing emotions from the user's voice and issuing an alert if the recognized emotion is abnormal.

[2062] Hardware and software used

[2063] Hardware:

[2064] server

[2065] Smartphone or smart glasses

[2066] microphone

[2067] software:

[2068] pyaudio library (audio recording)

[2069] wave library (storing audio data)

[2070] emotion_recognition library (emotion recognition)

[2071] System processing overview

[2072] Audio recording

[2073] The device (smartphone or smart glasses) records the audio during the meeting. It uses the pyaudio library to capture audio data in real time and save it in WAV format.

[2074] emotion recognition

[2075] The server receives the recorded audio data and analyzes the emotions using the emotion_recognition library, which analyzes the audio data and identifies the user's emotional state.

[2076] Information Extraction and Presentation

[2077] The server extracts information related to a specific content from the conference information, and the extracted information is presented to the user visually or audibly.

[2078] Alert occurs

[2079] The server issues an alert if the recognized emotion is abnormal (e.g., anger, fear, stress), which is then notified to the user or administrator.

[2080] Specific examples

[2081] For example, if a participant suddenly becomes angry during a meeting, the device will record the audio and send it to the server. The server will then analyze the recorded audio and recognize the emotion of "anger." If the recognized emotion is abnormal, the server will send an alert to the administrator and prompt them to take appropriate action.

[2082] Prompt Sentence Examples

[2083] Analyze the audio data recorded during the meeting and recognize the emotions of the participants. If emotions such as "anger," "fear," and "stress" are detected, an alert will be issued.

[2084] In this way, it is possible to grasp the emotions of participants in a meeting in real time, detect abnormal emotional changes, and take appropriate action.In addition, when extracting information related to specific content from recorded meeting data, user emotions can be taken into consideration, improving the quality and safety of meetings.

[2085] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2086] Step 1:

[2087] The terminal records audio during a conference. It uses the pyaudio library to capture audio data in real time and save it in WAV format. The input is the audio signal from the microphone, and the output is a WAV audio file. Specifically, the terminal reads the audio data at regular intervals and saves it in a buffer.

[2088] Step 2:

[2089] The terminal sends the recorded audio data to the server. The input is a WAV format audio file, and the output is the transfer of audio data to the server. Specifically, the terminal uploads the audio file to the server via the network.

[2090] Step 3:

[2091] The server analyzes the received audio data and recognizes emotions. It uses the emotion_recognition library to analyze the audio data and identify the user's emotional state. The input is a WAV audio file, and the output is the recognized emotion (e.g., anger, fear, stress). Specifically, the server inputs the audio data into the library and runs the emotion recognition model.

[2092] Step 4:

[2093] The server determines whether the recognized emotion is abnormal. The input is the recognized emotion, and the output is the abnormal emotion determination result. Specifically, the server compares the recognized emotion with a predefined list of abnormal emotions, and if there is a match, it determines that the emotion is abnormal.

[2094] Step 5:

[2095] The server issues an alert when an abnormal emotion is detected. The input is the abnormal emotion judgment result, and the output is an alert notification. Specifically, the server sends an email or push notification to the administrator to inform them that an abnormal emotion has been detected.

[2096] Step 6:

[2097] The server extracts information related to specific content from the meeting information. The input is the meeting information (video and audio recordings), and the output is the extracted information. Specifically, the server uses natural language processing technology to extract information related to specific topics or themes that interest the user from the meeting information.

[2098] Step 7:

[2099] The server presents the extracted information to the user. The input is the extracted information, and the output is the presentation of the information to the user. In concrete terms, the server displays the extracted information through an interface for providing it to the user visually or audibly.

[2100] Example 2

[2101] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2102] Conventional meeting information management systems have made it difficult for users to quickly search for specific content, especially when extracting information from audio data. Furthermore, they do not provide information that takes into account the user's emotions, which means they are unable to provide information that meets the user's needs.

[2103] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a plurality of pieces of conference information as input and extracting information related to specific content, means for presenting the extracted information to the user, means for receiving a voice inquiry from the user and converting the voice into text, means for extracting related information based on the converted text, and means for recognizing emotions from the user's voice and filtering information based on the emotions. This enables the user to quickly and accurately obtain information related to specific content, and further enables information to be provided according to the user's emotions.

[2104] "Meeting information" refers to data such as audio, video, and text recorded during a meeting.

[2105] "Information related to specific content" refers to information about specific topics or comments within the meeting information that is extracted based on a user's inquiry.

[2106] The term "extraction means" refers to a technique or method for extracting information related to specific content from the conference information.

[2107] "Means of presentation to the user" refers to the technology or method used to display or audibly convey the extracted information to the user.

[2108] "Voice inquiry" refers to a question or request made by a user to a system using voice.

[2109] "Means for converting speech to text" refers to the technology or method for converting speech data into text data.

[2110] "Means for extracting relevant information" refers to techniques and methods for extracting in...

Claims

1. A means of accepting inquiries from users; A means for extracting information related to the inquiry from video data or audio data representing a plurality of conferences or conversations using a prompt sentence instructing to extract information related to the inquiry from video data or audio data representing a plurality of conferences or conversations and a generative AI model; means for providing the extracted information related to the query to the user; A system including:

2. means for recognizing an emotion of the user using an emotion engine; 2. The system of claim 1, wherein the means for providing to the user provides the user with information related to the extracted query using a prompt sentence instructing the user to provide information related to the extracted query based on the recognized emotion of the user and the generative AI model.

3. The system of claim 1 , wherein the plurality of conferences or conversations are conducted in a vehicle.

Images