System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving and converting audio data into text data; means for analyzing the text data regarding a meeting and summarizing key points; means for performing sentiment analysis of the text data and providing the results thereof; and means for integrating the results of the analysis and providing the results to a user who participates late.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, the use of web conferences is increasing. However, overlapping meetings and lateness can make it difficult for participants to immediately grasp the progress of the meeting. This problem can reduce the efficiency and productivity of the meeting and cause stress to participants. The objective of this invention is to provide a system that allows late participants to quickly grasp the content of the meeting and smoothly join the discussion. [Means for solving the problem]

[0005] The present invention is a system that includes the following means: means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing the important points, means for performing sentiment analysis on the text data and providing the results, and means for integrating the analysis results and providing them to users who join late. This allows users to quickly grasp the progress and atmosphere of the meeting and efficiently participate in the discussion.

[0006] "Audio Data" means a digital recording of the voices and other audio information uttered by participants during a conference.

[0007] "Text data" refers to digital character information that has been converted from audio data into text.

[0008] "Analysis" is the process of processing audio and text data to extract information and structure the data.

[0009] "Key points" are statements, decisions, or key points of discussion that deserve particular attention in a meeting.

[0010] A "summary" is information that summarizes a longer text or conversation in a concise form.

[0011] "Sentiment analysis" is a technique that analyzes the emotional tone and mood of participants from text data and derives results.

[0012] "Means" are the techniques or methods used to achieve a particular purpose.

[0013] "Integration" refers to the process of bringing together multiple analysis results and data into a single entity.

[0014] "User" means a person who participates in a web conference using this system. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and participate in the discussion efficiently. This system analyzes voice and text data in real time and provides users with the necessary information.

[0037] Server Processing

[0038] Receiving and converting audio data

[0039] The server receives voice data sent from the device in real time. This voice data is passed to a voice recognition tool and converted into text data. This voice recognition tool uses the latest natural language processing technology to convert the voice data into text with high accuracy.

[0040] Summary of the discussion

[0041] The converted text data is input into a generative AI model on the server. This generative AI model analyzes the text data from the meeting, extracts and summarizes the key points and main points of the discussion, and provides this summary information to the user later.

[0042] sentiment analysis

[0043] The server then passes the converted text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0044] Generating and sending integration results

[0045] The server combines the summary information and sentiment analysis results into a single, cohesive dataset, which is then sent to the device in real time for immediate viewing by the user.

[0046] Terminal handling

[0047] Data capture and transmission

[0048] When a user joins a web conference late, the device captures the conference audio data in real time. This audio data is sent to the server in a stream format. At the same time, chat messages generated during the conference are also sent to the server.

[0049] Receiving and displaying analysis results

[0050] The analysis results sent from the server are instantly received by the device, which then displays the results in a user interface for easy user understanding.

[0051] User Actions

[0052] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[0053] Specific examples

[0054] In the case of a regular meeting of the sales department

[0055] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00. When the user joins at 10:15, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. The user can check this information on the device and immediately understand the progress of the meeting.

[0056] New product development meeting

[0057] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to identify key points and the emotions of the participants. The user, who joined late at 14:20, can quickly grasp the flow of the discussion and the opinions of the participants by checking the information displayed on his / her device.

[0058] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] The device verifies the user's authentication information for late joining a web conference, connects to the conference, and begins capturing the conference audio data in real time.

[0062] Step 2:

[0063] The device sends the captured audio data to the server in a stream format. This streaming is performed in real time to avoid delays. Chat messages are also sent immediately when they occur.

[0064] Step 3:

[0065] The server receives the voice data sent from the device and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the text data is temporarily stored.

[0066] Step 4:

[0067] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[0068] Step 5:

[0069] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[0070] Step 6:

[0071] The server combines the generated summary data with the results of the sentiment analysis to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[0072] Step 7:

[0073] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[0074] Step 8:

[0075] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[0076] Step 9:

[0077] Users can view summary information and sentiment analysis results displayed on their devices, allowing them to quickly grasp the progress of the meeting and the key points of the discussion.

[0078] Step 10:

[0079] Based on the information they have confirmed, users can express their opinions and ask questions at the appropriate time, allowing them to participate in meetings smoothly.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] Current web conferencing systems have the problem that late-joining users take time to understand the progress of the meeting and the content of the discussion, which reduces the efficiency of the meeting. In addition, there is a lack of means to grasp the emotional trends of participants, making it difficult to understand the atmosphere of the meeting and the emotional reactions of each participant. This makes it difficult for late-joining users to smoothly join the discussion.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, and means for performing sentiment analysis on the text data and providing the results, thereby enabling users who join late to quickly grasp the progress of the meeting, important points, and the emotional trends of the participants.

[0085] "Audio data" refers to audio information sent by users, and is data that includes the audio and speech of the conference.

[0086] "Text data" is voice data converted into text, and is data that expresses the contents of discussions and statements made at a meeting as character information.

[0087] "Analysis" refers to the act of processing received data, extracting important information, and analyzing it.

[0088] A "summary" refers to information that extracts important points from analyzed data and summarizes them concisely.

[0089] "Sentiment analysis" refers to the act of determining and analyzing participants' emotional responses from text data.

[0090] "Integration" refers to the act of combining different analytical data into a single coherent data set.

[0091] "User" refers to a person who participates in a web conference using this system.

[0092] A "generative AI model" is a trained artificial intelligence model that is capable of analyzing text data and generating summaries or other outputs.

[0093] A "prompt" is an input instruction to a generative AI model, and refers to the text provided to obtain the appropriate output.

[0094] "Real-time" refers to the responsiveness of data being processed immediately and provided to users the moment it is generated or transmitted.

[0095] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. The system of the present invention analyzes voice data and text data in real time and provides users with the necessary information.

[0096] Server configuration and processing

[0097] The server receives the voice data in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text API). This speech recognition tool uses the latest natural language processing technology and can convert voice data into text with high accuracy.

[0098] The server then uses a generative AI model (e.g., OpenAI GPT-3) to analyze the text data and summarize the key points of the meeting. By inputting prompts into this generative AI model, it is possible to extract the gist of the meeting discussion.

[0099] In addition, the server uses a sentiment analysis tool (e.g., IBM Watson Tone Analyzer) to analyze the atmosphere of the meeting and the emotions of the participants from the text data. The results of the sentiment analysis allow users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0100] Finally, the server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[0101] Terminal configuration and handling

[0102] When a user joins a web conference late, the device captures the conference audio data in real time and sends the audio data in a stream format to the server. Chat messages generated during the conference are also sent to the server. The device receives the analysis results sent from the server in real time and displays them on the user interface.

[0103] User operations

[0104] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[0105] Specific example explanation

[0106] For example, suppose a user joins a regular sales meeting late at 10:15. The meeting started at 10:00, and the server has received the voice and chat data up to that point. When the user joins the meeting, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. By checking this information, the user can immediately understand the progress of the meeting.

[0107] As another example, suppose a user joins a new product development meeting late at 2:20 PM. The meeting begins at 2:00 PM, and lively discussions are taking place about new effects. The server analyzes the voice and chat data to determine key points and participants' emotions. By checking the information displayed on the device, the user can quickly grasp the flow of the discussion and the opinions of the participants.

[0108] Example prompts for generative AI models

[0109] Here are some example prompts to input to a generative AI model:

[0110] Meeting text data:

[0111] 1. "Participant A: Regarding this project..."

[0112] 2. "Participant B: That's a great idea, but..."

[0113] 3. "Participant C: In response, I..."

[0114] Summarize the key points:

[0115] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[0116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0117] Step 1: Receiving and converting audio data

[0118] The server receives voice data sent from the device in real time. As input, it receives voice data from the device via WebSocket or RTMP protocol. This voice data is passed to a voice recognition tool (e.g., Google Cloud Speech-to-Text API) and converted into text data. Specifically, the server establishes a WebSocket connection and stores the voice data in a buffer. Once a certain amount of data has accumulated, it sends it to the voice recognition tool, which converts the voice data into text data. The converted text data is obtained as output.

[0119] Step 2: Summarize the discussion

[0120] The server inputs the converted speech-to-text data into a generative AI model (e.g., OpenAI GPT-3). As input, it cleans the converted text data and formats it into a format suitable for the generative AI model. It creates a prompt and supplies it to the generative AI model. Specifically, it formats the text data, inputs it into the generative AI model, and forms a prompt that generates a summary. As output, it obtains the summary data returned by the generative AI model.

[0121] Step 3: Sentiment analysis

[0122] The server passes the converted text data to a sentiment analysis tool (e.g., IBM Watson Tone Analyzer). As input, it sends the formatted text data to the sentiment analysis tool. Specifically, the server converts the text data into a format suitable for sentiment analysis and sends it to the sentiment analysis tool. It receives the results from the sentiment analysis tool and converts them into a format that is easy to analyze. As output, it obtains the sentiment analysis results.

[0123] Step 4: Generate and send integration results

[0124] The server integrates the summary information and the sentiment analysis results to form a single dataset. It receives the summary information and the sentiment analysis results as input. Specifically, the server integrates the summary information and the sentiment analysis results to generate a dataset in JSON format or similar. It then sends the integrated dataset to the device using WebSocket or HTTP POST. The integrated dataset is sent to the device as output.

[0125] Step 5: Capturing and Sending Data

[0126] The terminal captures the conference audio data in real time when a user joins a web conference late. As input, the terminal captures the conference audio data and sends it to the server in stream format. Specifically, the terminal starts audio capture when the user joins the conference application late. The terminal buffers a certain number of bytes of audio data and sends it to the server in stream format. As output, the captured audio data is sent to the server.

[0127] Step 6: Receive and display analysis results

[0128] The device receives the analysis results sent from the server in real time. As input, it monitors WebSocket connections and HTTP long polling to receive datasets from the server. Specific operations include analyzing the received datasets and displaying them in the user interface. The user can check the displayed information and chat or make comments as needed. As output, the user can check the progress of the meeting, summary information, and sentiment analysis results on the device.

[0129] By following these steps, even users who join late can quickly grasp the progress and emotional trends of the meeting and participate in the discussion efficiently.

[0130] (Application example 1)

[0131] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In conventional web conferencing systems, users who join a meeting late have difficulty quickly grasping the progress and atmosphere of the meeting and immediately and efficiently participating in the discussion. In addition, in order for users who join late to understand the overall picture and important points of the meeting, they have to ask other participants for explanations, which can reduce the productivity and efficiency of the meeting.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0134] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis on the text data and providing the results, means for integrating the analysis results and providing them to users joining late, and means for displaying the summary information of the meeting and the sentiment analysis results on a smartphone or a head-mounted display, thereby enabling users joining late to quickly grasp the progress of the meeting, important points, and the emotions of the participants, and to participate in the discussion efficiently.

[0135] "Audio data" refers to sound information, including speech and dialogue, that occurs during a meeting, and is digital data that is sent to a server for analysis.

[0136] "Text data" is voice data converted into text, and is data in which the contents of the conference and the contents of statements are expressed as character information.

[0137] "Means for receiving voice data and converting it into text data" refers to the process, technology, or software for receiving voice data during a conference on a server and converting it into text information.

[0138] A "means for summarizing key points" is an algorithm or technology that extracts important information and the gist of discussions from what is said during a meeting and summarizes them concisely.

[0139] "Means for performing sentiment analysis of text data and providing the results" refers to technology or software that analyzes text data to understand the emotional state of the speaker and the atmosphere of the meeting, and provides that information to users.

[0140] "Means for integrating analytical results and providing them to late-joining users" refers to technologies and interfaces that compile summarized information and sentiment analysis results into a single, cohesive data set and provide it to late-joining users in real time.

[0141] A "smartphone" is a type of mobile phone, a portable device that has advanced computing power and connectivity and can run various applications.

[0142] A "head-mounted display (HMD)" is a display device worn by the user on the head to provide visual information, and is also used to display information in meetings.

[0143] This invention is a system that allows users who join a web conference late to quickly understand the progress and atmosphere of the conference and efficiently participate in the discussion. Below, we will explain the processing content and details of the program that realizes the system.

[0144] Server Processing

[0145] Receiving and converting audio data

[0146] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using a voice recognition tool. For voice recognition, a common voice recognition tool such as Google Speech Recognition API is used.

[0147] Summary of the discussion

[0148] The converted text data is then input into a generative AI model, which uses a Japanese-specific model such as "rinna / japanese-gpt-1b" to analyze the meeting text data and extract and summarize key points and the gist of the discussion.

[0149] sentiment analysis

[0150] The server performs sentiment analysis on the converted text data using sentiment analysis tools such as "daigo / bert-base-japanese-sentiment." This analysis allows the user to understand the atmosphere of the meeting and the emotions of the participants.

[0151] Generating and sending integration results

[0152] The server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[0153] Terminal handling

[0154] Data capture and transmission

[0155] When a user joins a conference late, the user's device captures the conference's audio data in real time, and sends the audio data in a stream format to the server. At the same time, chat messages generated during the conference are also sent to the server.

[0156] Receiving and displaying analysis results

[0157] The analysis results sent from the server are instantly received on the user's device, which then displays the results in an interface that makes them easy to understand for the user, using a smartphone or head-mounted display (HMD).

[0158] User Actions

[0159] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can check the information displayed on their devices and make necessary comments or questions.

[0160] Specific examples

[0161] Example 1: Meeting about new product displays

[0162] For example, if a staff member is late to a meeting about displaying a new product, this system can immediately grasp the key points of the meeting and the opinions of the participants, making it possible to take appropriate action.

[0163] Example prompts for generative AI models

[0164] "Summarize the following text. Generate a summary of a maximum of 50 characters and a minimum of 25 characters."

[0165] text:

[0166] We are starting discussions about a new product. We are currently discussing how to improve the product's packaging design. There is a strong consensus that a design based on the color red would be good.

[0167] This allows staff who join late to quickly catch up on information and participate effectively in the meeting.

[0168] In this way, the system of the present invention allows even late-joining users to join the conference efficiently, improving the productivity and efficiency of the conference.

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1: Receiving and converting audio data

[0171] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using the Google Speech Recognition API. The input is voice data, and the output is recognized text data. In this conversion process, the voice waveform data is analyzed and each phoneme is converted into the corresponding text information.

[0172] Step 2: Summarize the discussion

[0173] The server inputs the text data obtained in step 1 into a generative AI model. This generative AI model uses "rinna / japanese-gpt-1b" to analyze the text data, extract key points and the gist of the argument, and generate a summary. The input is text data, and the output is a summary that summarizes the main points. Specifically, the model analyzes language patterns, identifies important keywords and phrases, and generates a short summary.

[0174] Step 3: Sentiment analysis

[0175] The server performs sentiment analysis on the text data obtained in step 1. For this analysis, it uses "daigo / bert-base-japanese-sentiment." The input is the text data, and the output is the sentiment analysis result (positive, negative, neutral, etc.). Specifically, the model extracts keywords and phrases related to emotions in the text, calculates each sentiment score, and performs classification.

[0176] Step 4: Integrating the analysis results

[0177] The server integrates the summary information obtained in step 2 with the sentiment analysis results obtained in step 3. The input is the summary sentence and the sentiment analysis results, and the output is an integrated dataset. In this integration process, each analysis result is associated and formatted in a user-friendly format.

[0178] Step 5: Viewing the analysis results

[0179] The user device receives the integrated dataset sent from the server in real time and displays it on a smartphone or head-mounted display (HMD). The input is the integrated dataset, and the output is the information displayed on the user interface. Specifically, summary information and sentiment analysis results are displayed on the device display, providing an interactive interface that allows the user to understand immediately.

[0180] Step 6: User Actions

[0181] Users check the information displayed on their devices to understand the progress of the meeting and key points. By making specific comments and asking questions, users can catch up and participate in the discussion efficiently. The input is the displayed information of the analysis results, and the output is the user's actions. Specifically, users operate the interface, obtain the necessary information, and make comments.

[0182] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0183] This invention relates to a system that enables users who join a web conference late to quickly grasp the progress and atmosphere of the conference, enabling them to participate in the discussion efficiently.Furthermore, by combining it with an emotion engine that recognizes the user's emotions and provides appropriate feedback based on those emotions, more advanced support is realized.

[0184] Server Processing

[0185] Receiving and converting audio data

[0186] The server receives the voice data sent from the device in real time and converts it into text data using a voice recognition tool. The converted text data is temporarily stored.

[0187] Summary of the discussion

[0188] The converted text data is then fed into a generative AI model hosted on a server, which extracts and summarizes the key points and main points of the meeting. This summary information is then provided to the user.

[0189] sentiment analysis

[0190] The server then passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0191] User emotion recognition

[0192] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[0193] Generating and sending integration results

[0194] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for immediate viewing by the user.

[0195] Terminal handling

[0196] Data capture and transmission

[0197] When a user joins a web conference late, the device captures the conference audio data in real time and transmits it to the server in a stream format. Chat messages that are generated are also transmitted to the server.

[0198] Receiving and displaying analysis results

[0199] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[0200] User Actions

[0201] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, the emotional trends of the participants, and feedback based on their own emotions, enabling them to participate in the discussion efficiently.

[0202] Specific examples

[0203] In the case of a regular meeting of the sales department

[0204] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins at 10:15, the server sends the summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[0205] New product development meeting

[0206] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes users' emotions in real time and generates feedback. The user, who joined late at 14:20, can smoothly participate in the discussion by checking the information displayed on his / her device, understanding the flow of the discussion, participants' opinions, and feedback based on his / her own emotions.

[0207] In this way, the system of the present invention not only enables users who join a web conference late to quickly and efficiently grasp the situation of the conference and participate in the discussion, but also improves overall conference productivity and participant satisfaction by recognizing users' emotions and providing appropriate feedback.

[0208] The processing flow will be explained below.

[0209] Step 1:

[0210] The device verifies the user's authentication information to join the web conference late and connects to the conference. At this time, it begins capturing the conference's audio data in real time. It also captures the user's facial expression data and voice tone and sends them to the server as emotional data.

[0211] Step 2:

[0212] The device transmits the captured audio data to the server in a stream format, with minimal latency and in real time, along with chat messages and emotional data.

[0213] Step 3:

[0214] The server receives the voice data sent from the terminal and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the converted text data is temporarily stored.

[0215] Step 4:

[0216] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[0217] Step 5:

[0218] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[0219] Step 6:

[0220] The server analyzes the user's emotional data sent from the device using an emotion engine. The emotion engine recognizes emotions in real time from the user's facial expressions and tone of voice. The recognized emotional data is reflected in the analysis results of the meeting.

[0221] Step 7:

[0222] The server combines the generated summary data, the results of the sentiment analysis, and the user sentiment data to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[0223] Step 8:

[0224] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[0225] Step 9:

[0226] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[0227] Step 10:

[0228] Users can view summary information and sentiment analysis results displayed on their devices, and also receive feedback based on their own emotional state, allowing them to quickly grasp the progress of the meeting, key points of discussion, participants' emotions, and their own state.

[0229] Step 11:

[0230] Based on the information they have checked, users can express their opinions and ask questions at the appropriate time, allowing them to participate smoothly in meetings. They can also refer to the displayed feedback and control their emotions while participating.

[0231] Example 2

[0232] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] In today's business environment, web conferencing is an important means of communication, but when users join late, it is difficult to grasp the situation and participate effectively in the discussion. Understanding the emotions of meeting participants is also essential, but there is a lack of ways to provide this information in real time. This makes it difficult for participants to smoothly join meetings without disrupting the flow of the discussion.

[0234] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis of the text data and providing the results, means for analyzing user sentiment and reflecting the data in the meeting analysis, and means for integrating the analysis results and providing them to users who join late. This makes it possible for users who join late to quickly and efficiently grasp the progress of the meeting and the sentiments of the participants, and actively participate in the discussion.

[0235] "Audio data" is data used to record or transmit sound in digital form.

[0236] "Text data" refers to data for recording linguistic information such as characters and symbols in digital form.

[0237] "Convert" refers to changing data of one format into data of another format.

[0238] "Summarizing" refers to extracting the key points of the original data and summarizing them in a short, concise form.

[0239] "Sentiment analysis" is a technology that analyzes emotional states from text data and audio data.

[0240] "Integrating" refers to bringing together multiple pieces of data or information into one coherent whole.

[0241] "User" refers to any individual or organization that uses the System.

[0242] A "means" refers to a device, method, or technique used to accomplish a particular purpose.

[0243] A "server" refers to a computer system that provides data and services to other computers (clients) on a computer network.

[0244] "Arriving late" means joining an event or activity (in this case a web conference) after the scheduled start time.

[0245] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. This system includes a series of processing steps: receiving and converting voice data, summarizing the discussion content, analyzing the sentiment of text data, recognizing users' emotions, and integrating and providing the analysis results.

[0246] The server receives the voice data from the device in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[0247] The server then analyzes the text data using a generative AI model (e.g., OpenAI's GPT-3) to summarize the key points and main points of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants. This allows the system to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0248] Furthermore, the server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. This engine includes technology that analyzes emotions from the user's voice and facial expressions. This emotion data is also reflected in the analysis results of the meeting.

[0249] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset. This integrated dataset is sent to the device in real time and can be viewed immediately by the user. By viewing this information, the user can quickly grasp the progress and key points of the meeting, the emotional trends of participants, and feedback based on their own emotions.

[0250] As a concrete example, suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins, the server sends a summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[0251] An example of a prompt is as follows:

[0252] "The regular sales meeting will start at 10:00. Please summarize the key points and the sentiment analysis results of the participants."

[0253] In the case of a new product development meeting, let's say a user joins late at 14:20. The meeting started at 14:00, and lively discussions about the new product were taking place. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes the user's emotions in real time and generates feedback. When a user joins late, they can smoothly participate in the discussion by checking the information displayed on their device, understanding the flow of the discussion, participants' opinions, and feedback based on their own emotions.

[0254] In this way, by using the system of this invention, users who join a web conference late can not only quickly and efficiently grasp the situation of the conference and participate in the discussion, but also recognize the user's emotions and provide appropriate feedback.

[0255] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0256] Step 1:

[0257] Capture and transmit audio data

[0258] When a user joins a web conference late, the device captures the conference's audio data in real time through the microphone, and transmits the captured audio data in a stream format to the server. Chat messages generated during the conference are also transmitted from the device to the server.

[0259] Input and Output

[0260] Input: Audio data obtained from the device's microphone, chat messages during the meeting

[0261] Output: Streamed audio data and chat messages sent to the server.

[0262] Specific actions

[0263] When a user launches a web conferencing app on their device, the microphone is automatically enabled and audio data is captured. The audio data is converted into a streaming format using a dedicated library and sent to the server via the HTTPS protocol. Chat messages are also uploaded to the server in their original format.

[0264] Step 2:

[0265] Receiving voice data and converting it to text

[0266] The server receives the voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[0267] Input and Output

[0268] Input: Stream format audio data sent from the device

[0269] Output: Text data generated by the speech recognition tool

[0270] Specific actions

[0271] After receiving the voice data, the server converts it into text using the Google Cloud Speech-to-Text API, which is then stored in a database.

[0272] Step 3:

[0273] Discussion summary and sentiment analysis

[0274] The server inputs the converted text data into a generative AI model (e.g., OpenAI GPT-3) to generate a summary of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants.

[0275] Input and Output

[0276] Input: Text data

[0277] Output: Summary information, sentiment analysis results

[0278] Specific actions

[0279] The server inputs the converted text data into a generative AI model via a Python script, generates summary information using prompt sentences, and passes the text data to a sentiment analysis tool to analyze participants' emotional states.

[0280] Step 4:

[0281] User emotion recognition

[0282] The server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. It analyzes emotions based on image data acquired from the user's camera and audio data acquired from the microphone, and reflects this emotional data in the meeting analysis.

[0283] Input and Output

[0284] Input: User's facial expression data, voice data

[0285] Output: User emotion data

[0286] Specific actions

[0287] The server periodically sends image and audio data captured by the user's camera to the emotion recognition engine, which analyzes it and outputs the user's emotional state. This data is then integrated with other analysis results.

[0288] Step 5:

[0289] Generating and sending integration results

[0290] The server combines the summary information, the results of sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for the user to view immediately.

[0291] Input and Output

[0292] Input: Summary information, sentiment analysis results, user sentiment data

[0293] Output: Unified dataset

[0294] Specific actions

[0295] The server aggregates information from multiple data sources and generates a single dataset using a Python script, which is then sent to the terminal via HTTPS.

[0296] Step 6:

[0297] Receiving and displaying analysis results

[0298] The terminal instantly receives the integrated result data sent from the server and displays it in real time on the user interface (UI), allowing users to check the progress of the meeting, key points, participants' emotions, and feedback based on their own emotions.

[0299] Input and Output

[0300] Input: Integrated dataset sent from the server

[0301] Output: Analysis results displayed in the user interface

[0302] Specific actions

[0303] The device receives the integrated dataset sent via HTTPS and displays the analysis results on the screen using a built-in UI framework.

[0304] Step 7:

[0305] Viewing and Working with Users

[0306] Even if a user joins a web conference late, they can quickly grasp the progress of the conference, important points, participants' emotional trends, and feedback based on their own emotions by referring to the information displayed in the UI, allowing them to participate in the discussion efficiently.

[0307] Input and Output

[0308] Input: Analysis results that users see on the screen

[0309] Output: User understanding and reference for efficient discussion participation

[0310] Specific actions

[0311] Users can scroll and click to view the information displayed on the UI and obtain the necessary information to participate in the meeting appropriately.

[0312] (Application example 2)

[0313] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0314] In today's world, when users arrive late to a product information session in a physical store, it is difficult for them to quickly grasp the progress and important points. Furthermore, there is a lack of systems that provide users with emotional feedback and support appropriate purchasing decisions. Therefore, there is a need for a method to improve the shopping experience and participation efficiency of product information sessions.

[0315] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing the important points, means for performing sentiment analysis of the text data and providing the results, means for recognizing the user's emotions and generating feedback based on the emotions, and means for integrating the analysis results and providing them to users who join late. This enables users who join late in a physical store to quickly understand the situation and receive appropriate feedback.

[0316] "Audio Data" means sound signals recorded by a microphone or other sound capture device.

[0317] "Text data" refers to document data obtained by converting voice data into text.

[0318] "Analysis" refers to the act of analyzing audio data or text data for a specific purpose.

[0319] "Summarization" refers to extracting important points from text data and generating a shortened document.

[0320] "Sentiment analysis" is a technique for identifying emotional states based on text and audio data.

[0321] "Feedback" refers to information or advice provided to users based on analysis results and sentiment analysis results.

[0322] "User emotion" refers to the emotional state of the user that can be derived from their voice and facial expressions.

[0323] "Integration" refers to the act of combining various analytical results and data into a single dataset.

[0324] "Real-time" refers to the timing of processing data and providing results immediately.

[0325] A "generative AI model" refers to an algorithm that uses machine learning to generate new information from data.

[0326] A "prompt sentence" is text that is used as input for an AI model.

[0327] This invention relates to a system that enables users who arrive late to product information sessions or events at physical stores to quickly grasp the progress and important points, and participate efficiently. It also recognizes the user's emotions and provides appropriate feedback based on those emotions.

[0328] Server Processing

[0329] The server receives the voice data in real time and converts it into text data using a speech recognition tool. Specifically, one available speech recognition tool is the speech_recognition library. The converted text data is temporarily saved. This text data is then input into a generative AI model, which summarizes the key points of the meeting and the main points of the discussion. This summary information is also later provided to the user. The generative AI model uses, for example, the Hugging Face transformers library.

[0330] Sentiment analysis is performed by passing text data to a sentiment analysis tool, such as the company's proprietary EmotionRecognizer module, which analyzes the atmosphere of the meeting and the emotions of the participants.

[0331] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[0332] Finally, the server combines the summary information, the sentiment analysis results, and the user's sentiment data to generate a single integrated dataset, which is sent to the device in real time for immediate user review.

[0333] Terminal handling

[0334] The device captures the audio data of a product presentation in a physical store in real time when the user joins late and sends it to the server in a stream format. Specific use cases include smartphones and smart glasses. Chat messages during the event are also sent to the server.

[0335] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[0336] Specific examples

[0337] Consider the example of a user arriving late to a new product briefing. Suppose the user joins 20 minutes after the briefing has started. The server receives and analyzes voice and chat data in real time from the start of the briefing. When the user joins, the server sends a summary of the briefing, sentiment analysis results, and feedback based on the user's sentiment to the device. The user can check this information, immediately understand the progress of the briefing, and receive appropriate feedback, allowing them to smoothly join the briefing.

[0338] Generative AI model prompt:

[0339] Please extract and summarize the following information from the audio data.

[0340] 1. Summary of the meeting

[0341] 2. Main opinions of participants

[0342] 3. Emotional atmosphere

[0343] This means that even if users arrive late, they can quickly obtain the information they need and participate in the product briefing efficiently.

[0344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0345] Step 1:

[0346] When a user joins a product briefing session at a physical store late, the device captures audio data of the session in real time. The captured audio data is sent to the server in stream format. In this step, audio data is recorded in real time using the device's microphone and sent to the server via the network. The input is the captured audio data, and the output is the audio data sent to the server.

[0347] Step 2:

[0348] The server converts the received voice data into text data in real time using a voice recognition tool. Specifically, it converts the voice data into text using the speech_recognition library. In this step, the input is voice data and the output is text data. The server temporarily stores the converted text data.

[0349] Step 3:

[0350] The server analyzes the converted text data using a generative AI model and summarizes the key points. In this step, the text data is input to the summarization model using Hugging Face's transformers library to obtain summarized text data. The input is text data, and the output is summarized text.

[0351] Step 4:

[0352] The server passes the summarized text data to the sentiment analysis tool to perform sentiment analysis on the data. Specifically, it uses the EmotionRecognizer module to recognize emotions in the text data and identify emotional states. In this step, the input is the summarized text and the output is the sentiment analysis results.

[0353] Step 5:

[0354] The server analyzes the user's voice and facial expressions to recognize the user's emotions. This involves analyzing voice data and image data using facial recognition technology. The server determines the user's emotional state based on this data. The input is the user's voice data and facial image data, and the output is the user's emotional state.

[0355] Step 6:

[0356] The server integrates the summary information, sentiment analysis results, and user emotion data to generate a single integrated dataset, which contains important information and emotional feedback that users need to understand. The inputs are the summary information, sentiment analysis results, and user emotion data, and the output is the integrated dataset.

[0357] Step 7:

[0358] The server transmits this integrated dataset to the terminal in real time. The transmitted data is immediately received by the terminal and displayed on the user interface. The input is the integrated dataset, and the output is the analysis results and feedback displayed on the terminal.

[0359] Step 8:

[0360] By checking the analysis results and feedback displayed on the device, users can grasp the progress of the product briefing, important points, and feedback based on their own emotions, allowing them to participate in the event efficiently. The input is the analysis results and feedback displayed on the device, and the output is improved user understanding and participation efficiency.

[0361] By following the above steps, users who arrive late at a store can quickly understand the situation and participate effectively.

[0362] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0363] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0364] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0365] [Second embodiment]

[0366] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0367] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0368] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0369] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0370] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0371] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0372] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0373] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0374] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0375] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0376] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0377] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0378] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and participate in the discussion efficiently. This system analyzes voice and text data in real time and provides users with the necessary information.

[0379] Server Processing

[0380] Receiving and converting audio data

[0381] The server receives voice data sent from the device in real time. This voice data is passed to a voice recognition tool and converted into text data. This voice recognition tool uses the latest natural language processing technology to convert the voice data into text with high accuracy.

[0382] Summary of the discussion

[0383] The converted text data is input into a generative AI model on the server. This generative AI model analyzes the text data from the meeting, extracts and summarizes the key points and main points of the discussion, and provides this summary information to the user later.

[0384] sentiment analysis

[0385] The server then passes the converted text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0386] Generating and sending integration results

[0387] The server combines the summary information and sentiment analysis results into a single, cohesive dataset, which is then sent to the device in real time for immediate viewing by the user.

[0388] Terminal handling

[0389] Data capture and transmission

[0390] When a user joins a web conference late, the device captures the conference audio data in real time. This audio data is sent to the server in a stream format. At the same time, chat messages generated during the conference are also sent to the server.

[0391] Receiving and displaying analysis results

[0392] The analysis results sent from the server are instantly received by the device, which then displays the results in a user interface for easy user understanding.

[0393] User Actions

[0394] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[0395] Specific examples

[0396] In the case of a regular meeting of the sales department

[0397] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00. When the user joins at 10:15, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. The user can check this information on the device and immediately understand the progress of the meeting.

[0398] New product development meeting

[0399] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to identify key points and the emotions of the participants. The user, who joined late at 14:20, can quickly grasp the flow of the discussion and the opinions of the participants by checking the information displayed on his / her device.

[0400] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[0401] The processing flow will be explained below.

[0402] Step 1:

[0403] The device verifies the user's authentication information for late joining a web conference, connects to the conference, and begins capturing the conference audio data in real time.

[0404] Step 2:

[0405] The device sends the captured audio data to the server in a stream format. This streaming is performed in real time to avoid delays. Chat messages are also sent immediately when they occur.

[0406] Step 3:

[0407] The server receives the voice data sent from the device and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the text data is temporarily stored.

[0408] Step 4:

[0409] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[0410] Step 5:

[0411] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[0412] Step 6:

[0413] The server combines the generated summary data with the results of the sentiment analysis to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[0414] Step 7:

[0415] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[0416] Step 8:

[0417] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[0418] Step 9:

[0419] Users can view summary information and sentiment analysis results displayed on their devices, allowing them to quickly grasp the progress of the meeting and the key points of the discussion.

[0420] Step 10:

[0421] Based on the information they have confirmed, users can express their opinions and ask questions at the appropriate time, allowing them to participate in meetings smoothly.

[0422] Example 1

[0423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] Current web conferencing systems have the problem that late-joining users take time to understand the progress of the meeting and the content of the discussion, which reduces the efficiency of the meeting. In addition, there is a lack of means to grasp the emotional trends of participants, making it difficult to understand the atmosphere of the meeting and the emotional reactions of each participant. This makes it difficult for late-joining users to smoothly join the discussion.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0426] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, and means for performing sentiment analysis on the text data and providing the results, thereby enabling users who join late to quickly grasp the progress of the meeting, important points, and the emotional trends of the participants.

[0427] "Audio data" refers to audio information sent by users, and is data that includes the audio and speech of the conference.

[0428] "Text data" is voice data converted into text, and is data that expresses the contents of discussions and statements made at a meeting as character information.

[0429] "Analysis" refers to the act of processing received data, extracting important information, and analyzing it.

[0430] A "summary" refers to information that extracts important points from analyzed data and summarizes them concisely.

[0431] "Sentiment analysis" refers to the act of determining and analyzing participants' emotional responses from text data.

[0432] "Integration" refers to the act of combining different analytical data into a single coherent data set.

[0433] "User" refers to a person who participates in a web conference using this system.

[0434] A "generative AI model" is a trained artificial intelligence model that is capable of analyzing text data and generating summaries or other outputs.

[0435] A "prompt" is an input instruction to a generative AI model, and refers to the text provided to obtain the appropriate output.

[0436] "Real-time" refers to the responsiveness of data being processed immediately and provided to users the moment it is generated or transmitted.

[0437] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. The system of the present invention analyzes voice data and text data in real time and provides users with the necessary information.

[0438] Server configuration and processing

[0439] The server receives the voice data in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text API). This speech recognition tool uses the latest natural language processing technology and can convert voice data into text with high accuracy.

[0440] The server then uses a generative AI model (e.g., OpenAI GPT-3) to analyze the text data and summarize the key points of the meeting. By inputting prompts into this generative AI model, it is possible to extract the gist of the meeting discussion.

[0441] In addition, the server uses a sentiment analysis tool (e.g., IBM Watson Tone Analyzer) to analyze the atmosphere of the meeting and the emotions of the participants from the text data. The results of the sentiment analysis allow users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0442] Finally, the server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[0443] Terminal configuration and handling

[0444] When a user joins a web conference late, the device captures the conference audio data in real time and sends the audio data in a stream format to the server. Chat messages generated during the conference are also sent to the server. The device receives the analysis results sent from the server in real time and displays them on the user interface.

[0445] User operations

[0446] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[0447] Specific example explanation

[0448] For example, suppose a user joins a regular sales meeting late at 10:15. The meeting started at 10:00, and the server has received the voice and chat data up to that point. When the user joins the meeting, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. By checking this information, the user can immediately understand the progress of the meeting.

[0449] As another example, suppose a user joins a new product development meeting late at 2:20 PM. The meeting begins at 2:00 PM, and lively discussions are taking place about new effects. The server analyzes the voice and chat data to determine key points and participants' emotions. By checking the information displayed on the device, the user can quickly grasp the flow of the discussion and the opinions of the participants.

[0450] Example prompts for generative AI models

[0451] Here are some example prompts to input to a generative AI model:

[0452] Meeting text data:

[0453] 1. "Participant A: Regarding this project..."

[0454] 2. "Participant B: That's a great idea, but..."

[0455] 3. "Participant C: In response, I..."

[0456] Summarize the key points:

[0457] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[0458] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0459] Step 1: Receiving and converting audio data

[0460] The server receives voice data sent from the device in real time. As input, it receives voice data from the device via WebSocket or RTMP protocol. This voice data is passed to a voice recognition tool (e.g., Google Cloud Speech-to-Text API) and converted into text data. Specifically, the server establishes a WebSocket connection and stores the voice data in a buffer. Once a certain amount of data has accumulated, it sends it to the voice recognition tool, which converts the voice data into text data. The converted text data is obtained as output.

[0461] Step 2: Summarize the discussion

[0462] The server inputs the converted speech-to-text data into a generative AI model (e.g., OpenAI GPT-3). As input, it cleans the converted text data and formats it into a format suitable for the generative AI model. It creates a prompt and supplies it to the generative AI model. Specifically, it formats the text data, inputs it into the generative AI model, and forms a prompt that generates a summary. As output, it obtains the summary data returned by the generative AI model.

[0463] Step 3: Sentiment analysis

[0464] The server passes the converted text data to a sentiment analysis tool (e.g., IBM Watson Tone Analyzer). As input, it sends the formatted text data to the sentiment analysis tool. Specifically, the server converts the text data into a format suitable for sentiment analysis and sends it to the sentiment analysis tool. It receives the results from the sentiment analysis tool and converts them into a format that is easy to analyze. As output, it obtains the sentiment analysis results.

[0465] Step 4: Generate and send integration results

[0466] The server integrates the summary information and the sentiment analysis results to form a single dataset. It receives the summary information and the sentiment analysis results as input. Specifically, the server integrates the summary information and the sentiment analysis results to generate a dataset in JSON format or similar. It then sends the integrated dataset to the device using WebSocket or HTTP POST. The integrated dataset is sent to the device as output.

[0467] Step 5: Capturing and Sending Data

[0468] The terminal captures the conference audio data in real time when a user joins a web conference late. As input, the terminal captures the conference audio data and sends it to the server in stream format. Specifically, the terminal starts audio capture when the user joins the conference application late. The terminal buffers a certain number of bytes of audio data and sends it to the server in stream format. As output, the captured audio data is sent to the server.

[0469] Step 6: Receive and display analysis results

[0470] The device receives the analysis results sent from the server in real time. As input, it monitors WebSocket connections and HTTP long polling to receive datasets from the server. Specific operations include analyzing the received datasets and displaying them in the user interface. The user can check the displayed information and chat or make comments as needed. As output, the user can check the progress of the meeting, summary information, and sentiment analysis results on the device.

[0471] By following these steps, even users who join late can quickly grasp the progress and emotional trends of the meeting and participate in the discussion efficiently.

[0472] (Application example 1)

[0473] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0474] In conventional web conferencing systems, users who join a meeting late have difficulty quickly grasping the progress and atmosphere of the meeting and immediately and efficiently participating in the discussion. In addition, in order for users who join late to understand the overall picture and important points of the meeting, they have to ask other participants for explanations, which can reduce the productivity and efficiency of the meeting.

[0475] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0476] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis on the text data and providing the results, means for integrating the analysis results and providing them to users joining late, and means for displaying the summary information of the meeting and the sentiment analysis results on a smartphone or a head-mounted display, thereby enabling users joining late to quickly grasp the progress of the meeting, important points, and the emotions of the participants, and to participate in the discussion efficiently.

[0477] "Audio data" refers to sound information, including speech and dialogue, that occurs during a meeting, and is digital data that is sent to a server for analysis.

[0478] "Text data" is voice data converted into text, and is data in which the contents of the conference and the contents of statements are expressed as character information.

[0479] "Means for receiving voice data and converting it into text data" refers to the process, technology, or software for receiving voice data during a conference on a server and converting it into text information.

[0480] A "means for summarizing key points" is an algorithm or technology that extracts important information and the gist of discussions from what is said during a meeting and summarizes them concisely.

[0481] "Means for performing sentiment analysis of text data and providing the results" refers to technology or software that analyzes text data to understand the emotional state of the speaker and the atmosphere of the meeting, and provides that information to users.

[0482] "Means for integrating analytical results and providing them to late-joining users" refers to technologies and interfaces that compile summarized information and sentiment analysis results into a single, cohesive data set and provide it to late-joining users in real time.

[0483] A "smartphone" is a type of mobile phone, a portable device that has advanced computing power and connectivity and can run various applications.

[0484] A "head-mounted display (HMD)" is a display device worn by the user on the head to provide visual information, and is also used to display information in meetings.

[0485] This invention is a system that allows users who join a web conference late to quickly understand the progress and atmosphere of the conference and efficiently participate in the discussion. Below, we will explain the processing content and details of the program that realizes the system.

[0486] Server Processing

[0487] Receiving and converting audio data

[0488] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using a voice recognition tool. For voice recognition, a common voice recognition tool such as Google Speech Recognition API is used.

[0489] Summary of the discussion

[0490] The converted text data is then input into a generative AI model, which uses a Japanese-specific model such as "rinna / japanese-gpt-1b" to analyze the meeting text data and extract and summarize key points and the gist of the discussion.

[0491] sentiment analysis

[0492] The server performs sentiment analysis on the converted text data using sentiment analysis tools such as "daigo / bert-base-japanese-sentiment." This analysis allows the user to understand the atmosphere of the meeting and the emotions of the participants.

[0493] Generating and sending integration results

[0494] The server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[0495] Terminal handling

[0496] Data capture and transmission

[0497] When a user joins a conference late, the user's device captures the conference's audio data in real time, and sends the audio data in a stream format to the server. At the same time, chat messages generated during the conference are also sent to the server.

[0498] Receiving and displaying analysis results

[0499] The analysis results sent from the server are instantly received on the user's device, which then displays the results in an interface that makes them easy to understand for the user, using a smartphone or head-mounted display (HMD).

[0500] User Actions

[0501] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can check the information displayed on their devices and make necessary comments or questions.

[0502] Specific examples

[0503] Example 1: Meeting about new product displays

[0504] For example, if a staff member is late to a meeting about displaying a new product, this system can immediately grasp the key points of the meeting and the opinions of the participants, making it possible to take appropriate action.

[0505] Example prompts for generative AI models

[0506] "Summarize the following text. Generate a summary of a maximum of 50 characters and a minimum of 25 characters."

[0507] text:

[0508] We are starting discussions about a new product. We are currently discussing how to improve the product's packaging design. There is a strong consensus that a design based on the color red would be good.

[0509] This allows staff who join late to quickly catch up on information and participate effectively in the meeting.

[0510] In this way, the system of the present invention allows even late-joining users to join the conference efficiently, improving the productivity and efficiency of the conference.

[0511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0512] Step 1: Receiving and converting audio data

[0513] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using the Google Speech Recognition API. The input is voice data, and the output is recognized text data. In this conversion process, the voice waveform data is analyzed and each phoneme is converted into the corresponding text information.

[0514] Step 2: Summarize the discussion

[0515] The server inputs the text data obtained in step 1 into a generative AI model. This generative AI model uses "rinna / japanese-gpt-1b" to analyze the text data, extract key points and the gist of the argument, and generate a summary. The input is text data, and the output is a summary that summarizes the main points. Specifically, the model analyzes language patterns, identifies important keywords and phrases, and generates a short summary.

[0516] Step 3: Sentiment analysis

[0517] The server performs sentiment analysis on the text data obtained in step 1. For this analysis, it uses "daigo / bert-base-japanese-sentiment." The input is the text data, and the output is the sentiment analysis result (positive, negative, neutral, etc.). Specifically, the model extracts keywords and phrases related to emotions in the text, calculates each sentiment score, and performs classification.

[0518] Step 4: Integrating the analysis results

[0519] The server integrates the summary information obtained in step 2 with the sentiment analysis results obtained in step 3. The input is the summary sentence and the sentiment analysis results, and the output is an integrated dataset. In this integration process, each analysis result is associated and formatted in a user-friendly format.

[0520] Step 5: Viewing the analysis results

[0521] The user device receives the integrated dataset sent from the server in real time and displays it on a smartphone or head-mounted display (HMD). The input is the integrated dataset, and the output is the information displayed on the user interface. Specifically, summary information and sentiment analysis results are displayed on the device display, providing an interactive interface that allows the user to understand immediately.

[0522] Step 6: User Actions

[0523] Users check the information displayed on their devices to understand the progress of the meeting and key points. By making specific comments and asking questions, users can catch up and participate in the discussion efficiently. The input is the displayed information of the analysis results, and the output is the user's actions. Specifically, users operate the interface, obtain the necessary information, and make comments.

[0524] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0525] This invention relates to a system that enables users who join a web conference late to quickly grasp the progress and atmosphere of the conference, enabling them to participate in the discussion efficiently.Furthermore, by combining it with an emotion engine that recognizes the user's emotions and provides appropriate feedback based on those emotions, more advanced support is realized.

[0526] Server Processing

[0527] Receiving and converting audio data

[0528] The server receives the voice data sent from the device in real time and converts it into text data using a voice recognition tool. The converted text data is temporarily stored.

[0529] Summary of the discussion

[0530] The converted text data is then fed into a generative AI model hosted on a server, which extracts and summarizes the key points and main points of the meeting. This summary information is then provided to the user.

[0531] sentiment analysis

[0532] The server then passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0533] User emotion recognition

[0534] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[0535] Generating and sending integration results

[0536] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for immediate viewing by the user.

[0537] Terminal handling

[0538] Data capture and transmission

[0539] When a user joins a web conference late, the device captures the conference audio data in real time and transmits it to the server in a stream format. Chat messages that are generated are also transmitted to the server.

[0540] Receiving and displaying analysis results

[0541] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[0542] User Actions

[0543] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, the emotional trends of the participants, and feedback based on their own emotions, enabling them to participate in the discussion efficiently.

[0544] Specific examples

[0545] In the case of a regular meeting of the sales department

[0546] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins at 10:15, the server sends the summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[0547] New product development meeting

[0548] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes users' emotions in real time and generates feedback. The user, who joined late at 14:20, can smoothly participate in the discussion by checking the information displayed on his / her device, understanding the flow of the discussion, participants' opinions, and feedback based on his / her own emotions.

[0549] In this way, the system of the present invention not only enables users who join a web conference late to quickly and efficiently grasp the situation of the conference and participate in the discussion, but also improves overall conference productivity and participant satisfaction by recognizing users' emotions and providing appropriate feedback.

[0550] The processing flow will be explained below.

[0551] Step 1:

[0552] The device verifies the user's authentication information to join the web conference late and connects to the conference. At this time, it begins capturing the conference's audio data in real time. It also captures the user's facial expression data and voice tone and sends them to the server as emotional data.

[0553] Step 2:

[0554] The device transmits the captured audio data to the server in a stream format, with minimal latency and in real time, along with chat messages and emotional data.

[0555] Step 3:

[0556] The server receives the voice data sent from the terminal and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the converted text data is temporarily stored.

[0557] Step 4:

[0558] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[0559] Step 5:

[0560] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[0561] Step 6:

[0562] The server analyzes the user's emotional data sent from the device using an emotion engine. The emotion engine recognizes emotions in real time from the user's facial expressions and tone of voice. The recognized emotional data is reflected in the analysis results of the meeting.

[0563] Step 7:

[0564] The server combines the generated summary data, the results of the sentiment analysis, and the user sentiment data to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[0565] Step 8:

[0566] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[0567] Step 9:

[0568] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[0569] Step 10:

[0570] Users can view summary information and sentiment analysis results displayed on their devices, and also receive feedback based on their own emotional state, allowing them to quickly grasp the progress of the meeting, key points of discussion, participants' emotions, and their own state.

[0571] Step 11:

[0572] Based on the information they have checked, users can express their opinions and ask questions at the appropriate time, allowing them to participate smoothly in meetings. They can also refer to the displayed feedback and control their emotions while participating.

[0573] Example 2

[0574] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0575] In today's business environment, web conferencing is an important means of communication, but when users join late, it is difficult to grasp the situation and participate effectively in the discussion. Understanding the emotions of meeting participants is also essential, but there is a lack of ways to provide this information in real time. This makes it difficult for participants to smoothly join meetings without disrupting the flow of the discussion.

[0576] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis of the text data and providing the results, means for analyzing user sentiment and reflecting the data in the meeting analysis, and means for integrating the analysis results and providing them to users who join late. This makes it possible for users who join late to quickly and efficiently grasp the progress of the meeting and the sentiments of the participants, and actively participate in the discussion.

[0577] "Audio data" is data used to record or transmit sound in digital form.

[0578] "Text data" refers to data for recording linguistic information such as characters and symbols in digital form.

[0579] "Convert" refers to changing data of one format into data of another format.

[0580] "Summarizing" refers to extracting the key points of the original data and summarizing them in a short, concise form.

[0581] "Sentiment analysis" is a technology that analyzes emotional states from text data and audio data.

[0582] "Integrating" refers to bringing together multiple pieces of data or information into one coherent whole.

[0583] "User" refers to any individual or organization that uses the System.

[0584] A "means" refers to a device, method, or technique used to accomplish a particular purpose.

[0585] A "server" refers to a computer system that provides data and services to other computers (clients) on a computer network.

[0586] "Arriving late" means joining an event or activity (in this case a web conference) after the scheduled start time.

[0587] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. This system includes a series of processing steps: receiving and converting voice data, summarizing the discussion content, analyzing the sentiment of text data, recognizing users' emotions, and integrating and providing the analysis results.

[0588] The server receives the voice data from the device in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[0589] The server then analyzes the text data using a generative AI model (e.g., OpenAI's GPT-3) to summarize the key points and main points of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants. This allows the system to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0590] Furthermore, the server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. This engine includes technology that analyzes emotions from the user's voice and facial expressions. This emotion data is also reflected in the analysis results of the meeting.

[0591] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset. This integrated dataset is sent to the device in real time and can be viewed immediately by the user. By viewing this information, the user can quickly grasp the progress and key points of the meeting, the emotional trends of participants, and feedback based on their own emotions.

[0592] As a concrete example, suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins, the server sends a summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[0593] An example of a prompt is as follows:

[0594] "The regular sales meeting will start at 10:00. Please summarize the key points and the sentiment analysis results of the participants."

[0595] In the case of a new product development meeting, let's say a user joins late at 14:20. The meeting started at 14:00, and lively discussions about the new product were taking place. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes the user's emotions in real time and generates feedback. When a user joins late, they can smoothly participate in the discussion by checking the information displayed on their device, understanding the flow of the discussion, participants' opinions, and feedback based on their own emotions.

[0596] In this way, by using the system of this invention, users who join a web conference late can not only quickly and efficiently grasp the situation of the conference and participate in the discussion, but also recognize the user's emotions and provide appropriate feedback.

[0597] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0598] Step 1:

[0599] Capture and transmit audio data

[0600] When a user joins a web conference late, the device captures the conference's audio data in real time through the microphone, and transmits the captured audio data in a stream format to the server. Chat messages generated during the conference are also transmitted from the device to the server.

[0601] Input and Output

[0602] Input: Audio data obtained from the device's microphone, chat messages during the meeting

[0603] Output: Streamed audio data and chat messages sent to the server.

[0604] Specific actions

[0605] When a user launches a web conferencing app on their device, the microphone is automatically enabled and audio data is captured. The audio data is converted into a streaming format using a dedicated library and sent to the server via the HTTPS protocol. Chat messages are also uploaded to the server in their original format.

[0606] Step 2:

[0607] Receiving voice data and converting it to text

[0608] The server receives the voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[0609] Input and Output

[0610] Input: Stream format audio data sent from the device

[0611] Output: Text data generated by the speech recognition tool

[0612] Specific actions

[0613] After receiving the voice data, the server converts it into text using the Google Cloud Speech-to-Text API, which is then stored in a database.

[0614] Step 3:

[0615] Discussion summary and sentiment analysis

[0616] The server inputs the converted text data into a generative AI model (e.g., OpenAI GPT-3) to generate a summary of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants.

[0617] Input and Output

[0618] Input: Text data

[0619] Output: Summary information, sentiment analysis results

[0620] Specific actions

[0621] The server inputs the converted text data into a generative AI model via a Python script, generates summary information using prompt sentences, and passes the text data to a sentiment analysis tool to analyze participants' emotional states.

[0622] Step 4:

[0623] User emotion recognition

[0624] The server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. It analyzes emotions based on image data acquired from the user's camera and audio data acquired from the microphone, and reflects this emotional data in the meeting analysis.

[0625] Input and Output

[0626] Input: User's facial expression data, voice data

[0627] Output: User emotion data

[0628] Specific actions

[0629] The server periodically sends image and audio data captured by the user's camera to the emotion recognition engine, which analyzes it and outputs the user's emotional state. This data is then integrated with other analysis results.

[0630] Step 5:

[0631] Generating and sending integration results

[0632] The server combines the summary information, the results of sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for the user to view immediately.

[0633] Input and Output

[0634] Input: Summary information, sentiment analysis results, user sentiment data

[0635] Output: Unified dataset

[0636] Specific actions

[0637] The server aggregates information from multiple data sources and generates a single dataset using a Python script, which is then sent to the terminal via HTTPS.

[0638] Step 6:

[0639] Receiving and displaying analysis results

[0640] The terminal instantly receives the integrated result data sent from the server and displays it in real time on the user interface (UI), allowing users to check the progress of the meeting, key points, participants' emotions, and feedback based on their own emotions.

[0641] Input and Output

[0642] Input: Integrated dataset sent from the server

[0643] Output: Analysis results displayed in the user interface

[0644] Specific actions

[0645] The device receives the integrated dataset sent via HTTPS and displays the analysis results on the screen using a built-in UI framework.

[0646] Step 7:

[0647] Viewing and Working with Users

[0648] Even if a user joins a web conference late, they can quickly grasp the progress of the conference, important points, participants' emotional trends, and feedback based on their own emotions by referring to the information displayed in the UI, allowing them to participate in the discussion efficiently.

[0649] Input and Output

[0650] Input: Analysis results that users see on the screen

[0651] Output: User understanding and reference for efficient discussion participation

[0652] Specific actions

[0653] Users can scroll and click to view the information displayed on the UI and obtain the necessary information to participate in the meeting appropriately.

[0654] (Application example 2)

[0655] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] In today's world, when users arrive late to a product information session in a physical store, it is difficult for them to quickly grasp the progress and important points. Furthermore, there is a lack of systems that provide users with emotional feedback and support appropriate purchasing decisions. Therefore, there is a need for a method to improve the shopping experience and participation efficiency of product information sessions.

[0657] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing the important points, means for performing sentiment analysis of the text data and providing the results, means for recognizing the user's emotions and generating feedback based on the emotions, and means for integrating the analysis results and providing them to users who join late. This enables users who join late in a physical store to quickly understand the situation and receive appropriate feedback.

[0658] "Audio Data" means sound signals recorded by a microphone or other sound capture device.

[0659] "Text data" refers to document data obtained by converting voice data into text.

[0660] "Analysis" refers to the act of analyzing audio data or text data for a specific purpose.

[0661] "Summarization" refers to extracting important points from text data and generating a shortened document.

[0662] "Sentiment analysis" is a technique for identifying emotional states based on text and audio data.

[0663] "Feedback" refers to information or advice provided to users based on analysis results and sentiment analysis results.

[0664] "User emotion" refers to the emotional state of the user that can be derived from their voice and facial expressions.

[0665] "Integration" refers to the act of combining various analytical results and data into a single dataset.

[0666] "Real-time" refers to the timing of processing data and providing results immediately.

[0667] A "generative AI model" refers to an algorithm that uses machine learning to generate new information from data.

[0668] A "prompt sentence" is text that is used as input for an AI model.

[0669] This invention relates to a system that enables users who arrive late to product information sessions or events at physical stores to quickly grasp the progress and important points, and participate efficiently. It also recognizes the user's emotions and provides appropriate feedback based on those emotions.

[0670] Server Processing

[0671] The server receives the voice data in real time and converts it into text data using a speech recognition tool. Specifically, one available speech recognition tool is the speech_recognition library. The converted text data is temporarily saved. This text data is then input into a generative AI model, which summarizes the key points of the meeting and the main points of the discussion. This summary information is also later provided to the user. The generative AI model uses, for example, the Hugging Face transformers library.

[0672] Sentiment analysis is performed by passing text data to a sentiment analysis tool, such as the company's proprietary EmotionRecognizer module, which analyzes the atmosphere of the meeting and the emotions of the participants.

[0673] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[0674] Finally, the server combines the summary information, the sentiment analysis results, and the user's sentiment data to generate a single integrated dataset, which is sent to the device in real time for immediate user review.

[0675] Terminal handling

[0676] The device captures the audio data of a product presentation in a physical store in real time when the user joins late and sends it to the server in a stream format. Specific use cases include smartphones and smart glasses. Chat messages during the event are also sent to the server.

[0677] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[0678] Specific examples

[0679] Consider the example of a user arriving late to a new product briefing. Suppose the user joins 20 minutes after the briefing has started. The server receives and analyzes voice and chat data in real time from the start of the briefing. When the user joins, the server sends a summary of the briefing, sentiment analysis results, and feedback based on the user's sentiment to the device. The user can check this information, immediately understand the progress of the briefing, and receive appropriate feedback, allowing them to smoothly join the briefing.

[0680] Generative AI model prompt:

[0681] Please extract and summarize the following information from the audio data.

[0682] 1. Summary of the meeting

[0683] 2. Main opinions of participants

[0684] 3. Emotional atmosphere

[0685] This means that even if users arrive late, they can quickly obtain the information they need and participate in the product briefing efficiently.

[0686] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0687] Step 1:

[0688] When a user joins a product briefing session at a physical store late, the device captures audio data of the session in real time. The captured audio data is sent to the server in stream format. In this step, audio data is recorded in real time using the device's microphone and sent to the server via the network. The input is the captured audio data, and the output is the audio data sent to the server.

[0689] Step 2:

[0690] The server converts the received voice data into text data in real time using a voice recognition tool. Specifically, it converts the voice data into text using the speech_recognition library. In this step, the input is voice data and the output is text data. The server temporarily stores the converted text data.

[0691] Step 3:

[0692] The server analyzes the converted text data using a generative AI model and summarizes the key points. In this step, the text data is input to the summarization model using Hugging Face's transformers library to obtain summarized text data. The input is text data, and the output is summarized text.

[0693] Step 4:

[0694] The server passes the summarized text data to the sentiment analysis tool to perform sentiment analysis on the data. Specifically, it uses the EmotionRecognizer module to recognize emotions in the text data and identify emotional states. In this step, the input is the summarized text and the output is the sentiment analysis results.

[0695] Step 5:

[0696] The server analyzes the user's voice and facial expressions to recognize the user's emotions. This involves analyzing voice data and image data using facial recognition technology. The server determines the user's emotional state based on this data. The input is the user's voice data and facial image data, and the output is the user's emotional state.

[0697] Step 6:

[0698] The server integrates the summary information, sentiment analysis results, and user emotion data to generate a single integrated dataset, which contains important information and emotional feedback that users need to understand. The inputs are the summary information, sentiment analysis results, and user emotion data, and the output is the integrated dataset.

[0699] Step 7:

[0700] The server transmits this integrated dataset to the terminal in real time. The transmitted data is immediately received by the terminal and displayed on the user interface. The input is the integrated dataset, and the output is the analysis results and feedback displayed on the terminal.

[0701] Step 8:

[0702] By checking the analysis results and feedback displayed on the device, users can grasp the progress of the product briefing, important points, and feedback based on their own emotions, allowing them to participate in the event efficiently. The input is the analysis results and feedback displayed on the device, and the output is improved user understanding and participation efficiency.

[0703] By following the above steps, users who arrive late at a store can quickly understand the situation and participate effectively.

[0704] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0705] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0706] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0707] [Third embodiment]

[0708] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0709] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0710] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0711] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0712] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0713] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0714] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0715] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0716] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0717] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0718] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0719] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0720] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and participate in the discussion efficiently. This system analyzes voice and text data in real time and provides users with the necessary information.

[0721] Server Processing

[0722] Receiving and converting audio data

[0723] The server receives voice data sent from the device in real time. This voice data is passed to a voice recognition tool and converted into text data. This voice recognition tool uses the latest natural language processing technology to convert the voice data into text with high accuracy.

[0724] Summary of the discussion

[0725] The converted text data is input into a generative AI model on the server. This generative AI model analyzes the text data from the meeting, extracts and summarizes the key points and main points of the discussion, and provides this summary information to the user later.

[0726] sentiment analysis

[0727] The server then passes the converted text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0728] Generating and sending integration results

[0729] The server combines the summary information and sentiment analysis results into a single, cohesive dataset, which is then sent to the device in real time for immediate viewing by the user.

[0730] Terminal handling

[0731] Data capture and transmission

[0732] When a user joins a web conference late, the device captures the conference audio data in real time. This audio data is sent to the server in a stream format. At the same time, chat messages generated during the conference are also sent to the server.

[0733] Receiving and displaying analysis results

[0734] The analysis results sent from the server are instantly received by the device, which then displays the results in a user interface for easy user understanding.

[0735] User Actions

[0736] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[0737] Specific examples

[0738] In the case of a regular meeting of the sales department

[0739] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00. When the user joins at 10:15, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. The user can check this information on the device and immediately understand the progress of the meeting.

[0740] New product development meeting

[0741] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to identify key points and the emotions of the participants. The user, who joined late at 14:20, can quickly grasp the flow of the discussion and the opinions of the participants by checking the information displayed on his / her device.

[0742] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[0743] The processing flow will be explained below.

[0744] Step 1:

[0745] The device verifies the user's authentication information for late joining a web conference, connects to the conference, and begins capturing the conference audio data in real time.

[0746] Step 2:

[0747] The device sends the captured audio data to the server in a stream format. This streaming is performed in real time to avoid delays. Chat messages are also sent immediately when they occur.

[0748] Step 3:

[0749] The server receives the voice data sent from the device and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the text data is temporarily stored.

[0750] Step 4:

[0751] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[0752] Step 5:

[0753] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[0754] Step 6:

[0755] The server combines the generated summary data with the results of the sentiment analysis to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[0756] Step 7:

[0757] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[0758] Step 8:

[0759] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[0760] Step 9:

[0761] Users can view summary information and sentiment analysis results displayed on their devices, allowing them to quickly grasp the progress of the meeting and the key points of the discussion.

[0762] Step 10:

[0763] Based on the information they have confirmed, users can express their opinions and ask questions at the appropriate time, allowing them to participate in meetings smoothly.

[0764] Example 1

[0765] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0766] Current web conferencing systems have the problem that late-joining users take time to understand the progress of the meeting and the content of the discussion, which reduces the efficiency of the meeting. In addition, there is a lack of means to grasp the emotional trends of participants, making it difficult to understand the atmosphere of the meeting and the emotional reactions of each participant. This makes it difficult for late-joining users to smoothly join the discussion.

[0767] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0768] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, and means for performing sentiment analysis on the text data and providing the results, thereby enabling users who join late to quickly grasp the progress of the meeting, important points, and the emotional trends of the participants.

[0769] "Audio data" refers to audio information sent by users, and is data that includes the audio and speech of the conference.

[0770] "Text data" is voice data converted into text, and is data that expresses the contents of discussions and statements made at a meeting as character information.

[0771] "Analysis" refers to the act of processing received data, extracting important information, and analyzing it.

[0772] A "summary" refers to information that extracts important points from analyzed data and summarizes them concisely.

[0773] "Sentiment analysis" refers to the act of determining and analyzing participants' emotional responses from text data.

[0774] "Integration" refers to the act of combining different analytical data into a single coherent data set.

[0775] "User" refers to a person who participates in a web conference using this system.

[0776] A "generative AI model" is a trained artificial intelligence model that is capable of analyzing text data and generating summaries or other outputs.

[0777] A "prompt" is an input instruction to a generative AI model, and refers to the text provided to obtain the appropriate output.

[0778] "Real-time" refers to the responsiveness of data being processed immediately and provided to users the moment it is generated or transmitted.

[0779] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. The system of the present invention analyzes voice data and text data in real time and provides users with the necessary information.

[0780] Server configuration and processing

[0781] The server receives the voice data in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text API). This speech recognition tool uses the latest natural language processing technology and can convert voice data into text with high accuracy.

[0782] The server then uses a generative AI model (e.g., OpenAI GPT-3) to analyze the text data and summarize the key points of the meeting. By inputting prompts into this generative AI model, it is possible to extract the gist of the meeting discussion.

[0783] In addition, the server uses a sentiment analysis tool (e.g., IBM Watson Tone Analyzer) to analyze the atmosphere of the meeting and the emotions of the participants from the text data. The results of the sentiment analysis allow users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0784] Finally, the server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[0785] Terminal configuration and handling

[0786] When a user joins a web conference late, the device captures the conference audio data in real time and sends the audio data in a stream format to the server. Chat messages generated during the conference are also sent to the server. The device receives the analysis results sent from the server in real time and displays them on the user interface.

[0787] User operations

[0788] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[0789] Specific example explanation

[0790] For example, suppose a user joins a regular sales meeting late at 10:15. The meeting started at 10:00, and the server has received the voice and chat data up to that point. When the user joins the meeting, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. By checking this information, the user can immediately understand the progress of the meeting.

[0791] As another example, suppose a user joins a new product development meeting late at 2:20 PM. The meeting begins at 2:00 PM, and lively discussions are taking place about new effects. The server analyzes the voice and chat data to determine key points and participants' emotions. By checking the information displayed on the device, the user can quickly grasp the flow of the discussion and the opinions of the participants.

[0792] Example prompts for generative AI models

[0793] Here are some example prompts to input to a generative AI model:

[0794] Meeting text data:

[0795] 1. "Participant A: Regarding this project..."

[0796] 2. "Participant B: That's a great idea, but..."

[0797] 3. "Participant C: In response, I..."

[0798] Summarize the key points:

[0799] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[0800] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0801] Step 1: Receiving and converting audio data

[0802] The server receives voice data sent from the device in real time. As input, it receives voice data from the device via WebSocket or RTMP protocol. This voice data is passed to a voice recognition tool (e.g., Google Cloud Speech-to-Text API) and converted into text data. Specifically, the server establishes a WebSocket connection and stores the voice data in a buffer. Once a certain amount of data has accumulated, it sends it to the voice recognition tool, which converts the voice data into text data. The converted text data is obtained as output.

[0803] Step 2: Summarize the discussion

[0804] The server inputs the converted speech-to-text data into a generative AI model (e.g., OpenAI GPT-3). As input, it cleans the converted text data and formats it into a format suitable for the generative AI model. It creates a prompt and supplies it to the generative AI model. Specifically, it formats the text data, inputs it into the generative AI model, and forms a prompt that generates a summary. As output, it obtains the summary data returned by the generative AI model.

[0805] Step 3: Sentiment analysis

[0806] The server passes the converted text data to a sentiment analysis tool (e.g., IBM Watson Tone Analyzer). As input, it sends the formatted text data to the sentiment analysis tool. Specifically, the server converts the text data into a format suitable for sentiment analysis and sends it to the sentiment analysis tool. It receives the results from the sentiment analysis tool and converts them into a format that is easy to analyze. As output, it obtains the sentiment analysis results.

[0807] Step 4: Generate and send integration results

[0808] The server integrates the summary information and the sentiment analysis results to form a single dataset. It receives the summary information and the sentiment analysis results as input. Specifically, the server integrates the summary information and the sentiment analysis results to generate a dataset in JSON format or similar. It then sends the integrated dataset to the device using WebSocket or HTTP POST. The integrated dataset is sent to the device as output.

[0809] Step 5: Capturing and Sending Data

[0810] The terminal captures the conference audio data in real time when a user joins a web conference late. As input, the terminal captures the conference audio data and sends it to the server in stream format. Specifically, the terminal starts audio capture when the user joins the conference application late. The terminal buffers a certain number of bytes of audio data and sends it to the server in stream format. As output, the captured audio data is sent to the server.

[0811] Step 6: Receive and display analysis results

[0812] The device receives the analysis results sent from the server in real time. As input, it monitors WebSocket connections and HTTP long polling to receive datasets from the server. Specific operations include analyzing the received datasets and displaying them in the user interface. The user can check the displayed information and chat or make comments as needed. As output, the user can check the progress of the meeting, summary information, and sentiment analysis results on the device.

[0813] By following these steps, even users who join late can quickly grasp the progress and emotional trends of the meeting and participate in the discussion efficiently.

[0814] (Application example 1)

[0815] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0816] In conventional web conferencing systems, users who join a meeting late have difficulty quickly grasping the progress and atmosphere of the meeting and immediately and efficiently participating in the discussion. In addition, in order for users who join late to understand the overall picture and important points of the meeting, they have to ask other participants for explanations, which can reduce the productivity and efficiency of the meeting.

[0817] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0818] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis on the text data and providing the results, means for integrating the analysis results and providing them to users joining late, and means for displaying the summary information of the meeting and the sentiment analysis results on a smartphone or a head-mounted display, thereby enabling users joining late to quickly grasp the progress of the meeting, important points, and the emotions of the participants, and to participate in the discussion efficiently.

[0819] "Audio data" refers to sound information, including speech and dialogue, that occurs during a meeting, and is digital data that is sent to a server for analysis.

[0820] "Text data" is voice data converted into text, and is data in which the contents of the conference and the contents of statements are expressed as character information.

[0821] "Means for receiving voice data and converting it into text data" refers to the process, technology, or software for receiving voice data during a conference on a server and converting it into text information.

[0822] A "means for summarizing key points" is an algorithm or technology that extracts important information and the gist of discussions from what is said during a meeting and summarizes them concisely.

[0823] "Means for performing sentiment analysis of text data and providing the results" refers to technology or software that analyzes text data to understand the emotional state of the speaker and the atmosphere of the meeting, and provides that information to users.

[0824] "Means for integrating analytical results and providing them to late-joining users" refers to technologies and interfaces that compile summarized information and sentiment analysis results into a single, cohesive data set and provide it to late-joining users in real time.

[0825] A "smartphone" is a type of mobile phone, a portable device that has advanced computing power and connectivity and can run various applications.

[0826] A "head-mounted display (HMD)" is a display device worn by the user on the head to provide visual information, and is also used to display information in meetings.

[0827] This invention is a system that allows users who join a web conference late to quickly understand the progress and atmosphere of the conference and efficiently participate in the discussion. Below, we will explain the processing content and details of the program that realizes the system.

[0828] Server Processing

[0829] Receiving and converting audio data

[0830] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using a voice recognition tool. For voice recognition, a common voice recognition tool such as Google Speech Recognition API is used.

[0831] Summary of the discussion

[0832] The converted text data is then input into a generative AI model, which uses a Japanese-specific model such as "rinna / japanese-gpt-1b" to analyze the meeting text data and extract and summarize key points and the gist of the discussion.

[0833] sentiment analysis

[0834] The server performs sentiment analysis on the converted text data using sentiment analysis tools such as "daigo / bert-base-japanese-sentiment." This analysis allows the user to understand the atmosphere of the meeting and the emotions of the participants.

[0835] Generating and sending integration results

[0836] The server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[0837] Terminal handling

[0838] Data capture and transmission

[0839] When a user joins a conference late, the user's device captures the conference's audio data in real time, and sends the audio data in a stream format to the server. At the same time, chat messages generated during the conference are also sent to the server.

[0840] Receiving and displaying analysis results

[0841] The analysis results sent from the server are instantly received on the user's device, which then displays the results in an interface that makes them easy to understand for the user, using a smartphone or head-mounted display (HMD).

[0842] User Actions

[0843] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can check the information displayed on their devices and make necessary comments or questions.

[0844] Specific examples

[0845] Example 1: Meeting about new product displays

[0846] For example, if a staff member is late to a meeting about displaying a new product, this system can immediately grasp the key points of the meeting and the opinions of the participants, making it possible to take appropriate action.

[0847] Example prompts for generative AI models

[0848] "Summarize the following text. Generate a summary of a maximum of 50 characters and a minimum of 25 characters."

[0849] text:

[0850] We are starting discussions about a new product. We are currently discussing how to improve the product's packaging design. There is a strong consensus that a design based on the color red would be good.

[0851] This allows staff who join late to quickly catch up on information and participate effectively in the meeting.

[0852] In this way, the system of the present invention allows even late-joining users to join the conference efficiently, improving the productivity and efficiency of the conference.

[0853] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0854] Step 1: Receiving and converting audio data

[0855] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using the Google Speech Recognition API. The input is voice data, and the output is recognized text data. In this conversion process, the voice waveform data is analyzed and each phoneme is converted into the corresponding text information.

[0856] Step 2: Summarize the discussion

[0857] The server inputs the text data obtained in step 1 into a generative AI model. This generative AI model uses "rinna / japanese-gpt-1b" to analyze the text data, extract key points and the gist of the argument, and generate a summary. The input is text data, and the output is a summary that summarizes the main points. Specifically, the model analyzes language patterns, identifies important keywords and phrases, and generates a short summary.

[0858] Step 3: Sentiment analysis

[0859] The server performs sentiment analysis on the text data obtained in step 1. For this analysis, it uses "daigo / bert-base-japanese-sentiment." The input is the text data, and the output is the sentiment analysis result (positive, negative, neutral, etc.). Specifically, the model extracts keywords and phrases related to emotions in the text, calculates each sentiment score, and performs classification.

[0860] Step 4: Integrating the analysis results

[0861] The server integrates the summary information obtained in step 2 with the sentiment analysis results obtained in step 3. The input is the summary sentence and the sentiment analysis results, and the output is an integrated dataset. In this integration process, each analysis result is associated and formatted in a user-friendly format.

[0862] Step 5: Viewing the analysis results

[0863] The user device receives the integrated dataset sent from the server in real time and displays it on a smartphone or head-mounted display (HMD). The input is the integrated dataset, and the output is the information displayed on the user interface. Specifically, summary information and sentiment analysis results are displayed on the device display, providing an interactive interface that allows the user to understand immediately.

[0864] Step 6: User Actions

[0865] Users check the information displayed on their devices to understand the progress of the meeting and key points. By making specific comments and asking questions, users can catch up and participate in the discussion efficiently. The input is the displayed information of the analysis results, and the output is the user's actions. Specifically, users operate the interface, obtain the necessary information, and make comments.

[0866] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0867] This invention relates to a system that enables users who join a web conference late to quickly grasp the progress and atmosphere of the conference, enabling them to participate in the discussion efficiently.Furthermore, by combining it with an emotion engine that recognizes the user's emotions and provides appropriate feedback based on those emotions, more advanced support is realized.

[0868] Server Processing

[0869] Receiving and converting audio data

[0870] The server receives the voice data sent from the device in real time and converts it into text data using a voice recognition tool. The converted text data is temporarily stored.

[0871] Summary of the discussion

[0872] The converted text data is then fed into a generative AI model hosted on a server, which extracts and summarizes the key points and main points of the meeting. This summary information is then provided to the user.

[0873] sentiment analysis

[0874] The server then passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0875] User emotion recognition

[0876] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[0877] Generating and sending integration results

[0878] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for immediate viewing by the user.

[0879] Terminal handling

[0880] Data capture and transmission

[0881] When a user joins a web conference late, the device captures the conference audio data in real time and transmits it to the server in a stream format. Chat messages that are generated are also transmitted to the server.

[0882] Receiving and displaying analysis results

[0883] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[0884] User Actions

[0885] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, the emotional trends of the participants, and feedback based on their own emotions, enabling them to participate in the discussion efficiently.

[0886] Specific examples

[0887] In the case of a regular meeting of the sales department

[0888] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins at 10:15, the server sends the summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[0889] New product development meeting

[0890] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes users' emotions in real time and generates feedback. The user, who joined late at 14:20, can smoothly participate in the discussion by checking the information displayed on his / her device, understanding the flow of the discussion, participants' opinions, and feedback based on his / her own emotions.

[0891] In this way, the system of the present invention not only enables users who join a web conference late to quickly and efficiently grasp the situation of the conference and participate in the discussion, but also improves overall conference productivity and participant satisfaction by recognizing users' emotions and providing appropriate feedback.

[0892] The processing flow will be explained below.

[0893] Step 1:

[0894] The device verifies the user's authentication information to join the web conference late and connects to the conference. At this time, it begins capturing the conference's audio data in real time. It also captures the user's facial expression data and voice tone and sends them to the server as emotional data.

[0895] Step 2:

[0896] The device transmits the captured audio data to the server in a stream format, with minimal latency and in real time, along with chat messages and emotional data.

[0897] Step 3:

[0898] The server receives the voice data sent from the terminal and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the converted text data is temporarily stored.

[0899] Step 4:

[0900] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[0901] Step 5:

[0902] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[0903] Step 6:

[0904] The server analyzes the user's emotional data sent from the device using an emotion engine. The emotion engine recognizes emotions in real time from the user's facial expressions and tone of voice. The recognized emotional data is reflected in the analysis results of the meeting.

[0905] Step 7:

[0906] The server combines the generated summary data, the results of the sentiment analysis, and the user sentiment data to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[0907] Step 8:

[0908] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[0909] Step 9:

[0910] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[0911] Step 10:

[0912] Users can view summary information and sentiment analysis results displayed on their devices, and also receive feedback based on their own emotional state, allowing them to quickly grasp the progress of the meeting, key points of discussion, participants' emotions, and their own state.

[0913] Step 11:

[0914] Based on the information they have checked, users can express their opinions and ask questions at the appropriate time, allowing them to participate smoothly in meetings. They can also refer to the displayed feedback and control their emotions while participating.

[0915] Example 2

[0916] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0917] In today's business environment, web conferencing is an important means of communication, but when users join late, it is difficult to grasp the situation and participate effectively in the discussion. Understanding the emotions of meeting participants is also essential, but there is a lack of ways to provide this information in real time. This makes it difficult for participants to smoothly join meetings without disrupting the flow of the discussion.

[0918] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis of the text data and providing the results, means for analyzing user sentiment and reflecting the data in the meeting analysis, and means for integrating the analysis results and providing them to users who join late. This makes it possible for users who join late to quickly and efficiently grasp the progress of the meeting and the sentiments of the participants, and actively participate in the discussion.

[0919] "Audio data" is data used to record or transmit sound in digital form.

[0920] "Text data" refers to data for recording linguistic information such as characters and symbols in digital form.

[0921] "Convert" refers to changing data of one format into data of another format.

[0922] "Summarizing" refers to extracting the key points of the original data and summarizing them in a short, concise form.

[0923] "Sentiment analysis" is a technology that analyzes emotional states from text data and audio data.

[0924] "Integrating" refers to bringing together multiple pieces of data or information into one coherent whole.

[0925] "User" refers to any individual or organization that uses the System.

[0926] A "means" refers to a device, method, or technique used to accomplish a particular purpose.

[0927] A "server" refers to a computer system that provides data and services to other computers (clients) on a computer network.

[0928] "Arriving late" means joining an event or activity (in this case a web conference) after the scheduled start time.

[0929] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. This system includes a series of processing steps: receiving and converting voice data, summarizing the discussion content, analyzing the sentiment of text data, recognizing users' emotions, and integrating and providing the analysis results.

[0930] The server receives the voice data from the device in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[0931] The server then analyzes the text data using a generative AI model (e.g., OpenAI's GPT-3) to summarize the key points and main points of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants. This allows the system to understand not only the progress of the meeting but also the emotional reactions of the participants.

[0932] Furthermore, the server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. This engine includes technology that analyzes emotions from the user's voice and facial expressions. This emotion data is also reflected in the analysis results of the meeting.

[0933] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset. This integrated dataset is sent to the device in real time and can be viewed immediately by the user. By viewing this information, the user can quickly grasp the progress and key points of the meeting, the emotional trends of participants, and feedback based on their own emotions.

[0934] As a concrete example, suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins, the server sends a summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[0935] An example of a prompt is as follows:

[0936] "The regular sales meeting will start at 10:00. Please summarize the key points and the sentiment analysis results of the participants."

[0937] In the case of a new product development meeting, let's say a user joins late at 14:20. The meeting started at 14:00, and lively discussions about the new product were taking place. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes the user's emotions in real time and generates feedback. When a user joins late, they can smoothly participate in the discussion by checking the information displayed on their device, understanding the flow of the discussion, participants' opinions, and feedback based on their own emotions.

[0938] In this way, by using the system of this invention, users who join a web conference late can not only quickly and efficiently grasp the situation of the conference and participate in the discussion, but also recognize the user's emotions and provide appropriate feedback.

[0939] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0940] Step 1:

[0941] Capture and transmit audio data

[0942] When a user joins a web conference late, the device captures the conference's audio data in real time through the microphone, and transmits the captured audio data in a stream format to the server. Chat messages generated during the conference are also transmitted from the device to the server.

[0943] Input and Output

[0944] Input: Audio data obtained from the device's microphone, chat messages during the meeting

[0945] Output: Streamed audio data and chat messages sent to the server.

[0946] Specific actions

[0947] When a user launches a web conferencing app on their device, the microphone is automatically enabled and audio data is captured. The audio data is converted into a streaming format using a dedicated library and sent to the server via the HTTPS protocol. Chat messages are also uploaded to the server in their original format.

[0948] Step 2:

[0949] Receiving voice data and converting it to text

[0950] The server receives the voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[0951] Input and Output

[0952] Input: Stream format audio data sent from the device

[0953] Output: Text data generated by the speech recognition tool

[0954] Specific actions

[0955] After receiving the voice data, the server converts it into text using the Google Cloud Speech-to-Text API, which is then stored in a database.

[0956] Step 3:

[0957] Discussion summary and sentiment analysis

[0958] The server inputs the converted text data into a generative AI model (e.g., OpenAI GPT-3) to generate a summary of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants.

[0959] Input and Output

[0960] Input: Text data

[0961] Output: Summary information, sentiment analysis results

[0962] Specific actions

[0963] The server inputs the converted text data into a generative AI model via a Python script, generates summary information using prompt sentences, and passes the text data to a sentiment analysis tool to analyze participants' emotional states.

[0964] Step 4:

[0965] User emotion recognition

[0966] The server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. It analyzes emotions based on image data acquired from the user's camera and audio data acquired from the microphone, and reflects this emotional data in the meeting analysis.

[0967] Input and Output

[0968] Input: User's facial expression data, voice data

[0969] Output: User emotion data

[0970] Specific actions

[0971] The server periodically sends image and audio data captured by the user's camera to the emotion recognition engine, which analyzes it and outputs the user's emotional state. This data is then integrated with other analysis results.

[0972] Step 5:

[0973] Generating and sending integration results

[0974] The server combines the summary information, the results of sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for the user to view immediately.

[0975] Input and Output

[0976] Input: Summary information, sentiment analysis results, user sentiment data

[0977] Output: Unified dataset

[0978] Specific actions

[0979] The server aggregates information from multiple data sources and generates a single dataset using a Python script, which is then sent to the terminal via HTTPS.

[0980] Step 6:

[0981] Receiving and displaying analysis results

[0982] The terminal instantly receives the integrated result data sent from the server and displays it in real time on the user interface (UI), allowing users to check the progress of the meeting, key points, participants' emotions, and feedback based on their own emotions.

[0983] Input and Output

[0984] Input: Integrated dataset sent from the server

[0985] Output: Analysis results displayed in the user interface

[0986] Specific actions

[0987] The device receives the integrated dataset sent via HTTPS and displays the analysis results on the screen using a built-in UI framework.

[0988] Step 7:

[0989] Viewing and Working with Users

[0990] Even if a user joins a web conference late, they can quickly grasp the progress of the conference, important points, participants' emotional trends, and feedback based on their own emotions by referring to the information displayed in the UI, allowing them to participate in the discussion efficiently.

[0991] Input and Output

[0992] Input: Analysis results that users see on the screen

[0993] Output: User understanding and reference for efficient discussion participation

[0994] Specific actions

[0995] Users can scroll and click to view the information displayed on the UI and obtain the necessary information to participate in the meeting appropriately.

[0996] (Application example 2)

[0997] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0998] In today's world, when users arrive late to a product information session in a physical store, it is difficult for them to quickly grasp the progress and important points. Furthermore, there is a lack of systems that provide users with emotional feedback and support appropriate purchasing decisions. Therefore, there is a need for a method to improve the shopping experience and participation efficiency of product information sessions.

[0999] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing the important points, means for performing sentiment analysis of the text data and providing the results, means for recognizing the user's emotions and generating feedback based on the emotions, and means for integrating the analysis results and providing them to users who join late. This enables users who join late in a physical store to quickly understand the situation and receive appropriate feedback.

[1000] "Audio Data" means sound signals recorded by a microphone or other sound capture device.

[1001] "Text data" refers to document data obtained by converting voice data into text.

[1002] "Analysis" refers to the act of analyzing audio data or text data for a specific purpose.

[1003] "Summarization" refers to extracting important points from text data and generating a shortened document.

[1004] "Sentiment analysis" is a technique for identifying emotional states based on text and audio data.

[1005] "Feedback" refers to information or advice provided to users based on analysis results and sentiment analysis results.

[1006] "User emotion" refers to the emotional state of the user that can be derived from their voice and facial expressions.

[1007] "Integration" refers to the act of combining various analytical results and data into a single dataset.

[1008] "Real-time" refers to the timing of processing data and providing results immediately.

[1009] A "generative AI model" refers to an algorithm that uses machine learning to generate new information from data.

[1010] A "prompt sentence" is text that is used as input for an AI model.

[1011] This invention relates to a system that enables users who arrive late to product information sessions or events at physical stores to quickly grasp the progress and important points, and participate efficiently. It also recognizes the user's emotions and provides appropriate feedback based on those emotions.

[1012] Server Processing

[1013] The server receives the voice data in real time and converts it into text data using a speech recognition tool. Specifically, one available speech recognition tool is the speech_recognition library. The converted text data is temporarily saved. This text data is then input into a generative AI model, which summarizes the key points of the meeting and the main points of the discussion. This summary information is also later provided to the user. The generative AI model uses, for example, the Hugging Face transformers library.

[1014] Sentiment analysis is performed by passing text data to a sentiment analysis tool, such as the company's proprietary EmotionRecognizer module, which analyzes the atmosphere of the meeting and the emotions of the participants.

[1015] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[1016] Finally, the server combines the summary information, the sentiment analysis results, and the user's sentiment data to generate a single integrated dataset, which is sent to the device in real time for immediate user review.

[1017] Terminal handling

[1018] The device captures the audio data of a product presentation in a physical store in real time when the user joins late and sends it to the server in a stream format. Specific use cases include smartphones and smart glasses. Chat messages during the event are also sent to the server.

[1019] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[1020] Specific examples

[1021] Consider the example of a user arriving late to a new product briefing. Suppose the user joins 20 minutes after the briefing has started. The server receives and analyzes voice and chat data in real time from the start of the briefing. When the user joins, the server sends a summary of the briefing, sentiment analysis results, and feedback based on the user's sentiment to the device. The user can check this information, immediately understand the progress of the briefing, and receive appropriate feedback, allowing them to smoothly join the briefing.

[1022] Generative AI model prompt:

[1023] Please extract and summarize the following information from the audio data.

[1024] 1. Summary of the meeting

[1025] 2. Main opinions of participants

[1026] 3. Emotional atmosphere

[1027] This means that even if users arrive late, they can quickly obtain the information they need and participate in the product briefing efficiently.

[1028] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1029] Step 1:

[1030] When a user joins a product briefing session at a physical store late, the device captures audio data of the session in real time. The captured audio data is sent to the server in stream format. In this step, audio data is recorded in real time using the device's microphone and sent to the server via the network. The input is the captured audio data, and the output is the audio data sent to the server.

[1031] Step 2:

[1032] The server converts the received voice data into text data in real time using a voice recognition tool. Specifically, it converts the voice data into text using the speech_recognition library. In this step, the input is voice data and the output is text data. The server temporarily stores the converted text data.

[1033] Step 3:

[1034] The server analyzes the converted text data using a generative AI model and summarizes the key points. In this step, the text data is input to the summarization model using Hugging Face's transformers library to obtain summarized text data. The input is text data, and the output is summarized text.

[1035] Step 4:

[1036] The server passes the summarized text data to the sentiment analysis tool to perform sentiment analysis on the data. Specifically, it uses the EmotionRecognizer module to recognize emotions in the text data and identify emotional states. In this step, the input is the summarized text and the output is the sentiment analysis results.

[1037] Step 5:

[1038] The server analyzes the user's voice and facial expressions to recognize the user's emotions. This involves analyzing voice data and image data using facial recognition technology. The server determines the user's emotional state based on this data. The input is the user's voice data and facial image data, and the output is the user's emotional state.

[1039] Step 6:

[1040] The server integrates the summary information, sentiment analysis results, and user emotion data to generate a single integrated dataset, which contains important information and emotional feedback that users need to understand. The inputs are the summary information, sentiment analysis results, and user emotion data, and the output is the integrated dataset.

[1041] Step 7:

[1042] The server transmits this integrated dataset to the terminal in real time. The transmitted data is immediately received by the terminal and displayed on the user interface. The input is the integrated dataset, and the output is the analysis results and feedback displayed on the terminal.

[1043] Step 8:

[1044] By checking the analysis results and feedback displayed on the device, users can grasp the progress of the product briefing, important points, and feedback based on their own emotions, allowing them to participate in the event efficiently. The input is the analysis results and feedback displayed on the device, and the output is improved user understanding and participation efficiency.

[1045] By following the above steps, users who arrive late at a store can quickly understand the situation and participate effectively.

[1046] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1047] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1048] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1049] [Fourth embodiment]

[1050] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1051] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1052] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1053] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1054] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1055] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1056] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1057] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1058] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1059] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1060] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1061] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1062] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1063] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and participate in the discussion efficiently. This system analyzes voice and text data in real time and provides users with the necessary information.

[1064] Server Processing

[1065] Receiving and converting audio data

[1066] The server receives voice data sent from the device in real time. This voice data is passed to a voice recognition tool and converted into text data. This voice recognition tool uses the latest natural language processing technology to convert the voice data into text with high accuracy.

[1067] Summary of the discussion

[1068] The converted text data is input into a generative AI model on the server. This generative AI model analyzes the text data from the meeting, extracts and summarizes the key points and main points of the discussion, and provides this summary information to the user later.

[1069] sentiment analysis

[1070] The server then passes the converted text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[1071] Generating and sending integration results

[1072] The server combines the summary information and sentiment analysis results into a single, cohesive dataset, which is then sent to the device in real time for immediate viewing by the user.

[1073] Terminal handling

[1074] Data capture and transmission

[1075] When a user joins a web conference late, the device captures the conference audio data in real time. This audio data is sent to the server in a stream format. At the same time, chat messages generated during the conference are also sent to the server.

[1076] Receiving and displaying analysis results

[1077] The analysis results sent from the server are instantly received by the device, which then displays the results in a user interface for easy user understanding.

[1078] User Actions

[1079] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[1080] Specific examples

[1081] In the case of a regular meeting of the sales department

[1082] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00. When the user joins at 10:15, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. The user can check this information on the device and immediately understand the progress of the meeting.

[1083] New product development meeting

[1084] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to identify key points and the emotions of the participants. The user, who joined late at 14:20, can quickly grasp the flow of the discussion and the opinions of the participants by checking the information displayed on his / her device.

[1085] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[1086] The processing flow will be explained below.

[1087] Step 1:

[1088] The device verifies the user's authentication information for late joining a web conference, connects to the conference, and begins capturing the conference audio data in real time.

[1089] Step 2:

[1090] The device sends the captured audio data to the server in a stream format. This streaming is performed in real time to avoid delays. Chat messages are also sent immediately when they occur.

[1091] Step 3:

[1092] The server receives the voice data sent from the device and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the text data is temporarily stored.

[1093] Step 4:

[1094] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[1095] Step 5:

[1096] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[1097] Step 6:

[1098] The server combines the generated summary data with the results of the sentiment analysis to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[1099] Step 7:

[1100] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[1101] Step 8:

[1102] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[1103] Step 9:

[1104] Users can view summary information and sentiment analysis results displayed on their devices, allowing them to quickly grasp the progress of the meeting and the key points of the discussion.

[1105] Step 10:

[1106] Based on the information they have confirmed, users can express their opinions and ask questions at the appropriate time, allowing them to participate in meetings smoothly.

[1107] Example 1

[1108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1109] Current web conferencing systems have the problem that late-joining users take time to understand the progress of the meeting and the content of the discussion, which reduces the efficiency of the meeting. In addition, there is a lack of means to grasp the emotional trends of participants, making it difficult to understand the atmosphere of the meeting and the emotional reactions of each participant. This makes it difficult for late-joining users to smoothly join the discussion.

[1110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1111] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, and means for performing sentiment analysis on the text data and providing the results, thereby enabling users who join late to quickly grasp the progress of the meeting, important points, and the emotional trends of the participants.

[1112] "Audio data" refers to audio information sent by users, and is data that includes the audio and speech of the conference.

[1113] "Text data" is voice data converted into text, and is data that expresses the contents of discussions and statements made at a meeting as character information.

[1114] "Analysis" refers to the act of processing received data, extracting important information, and analyzing it.

[1115] A "summary" refers to information that extracts important points from analyzed data and summarizes them concisely.

[1116] "Sentiment analysis" refers to the act of determining and analyzing participants' emotional responses from text data.

[1117] "Integration" refers to the act of combining different analytical data into a single coherent data set.

[1118] "User" refers to a person who participates in a web conference using this system.

[1119] A "generative AI model" is a trained artificial intelligence model that is capable of analyzing text data and generating summaries or other outputs.

[1120] A "prompt" is an input instruction to a generative AI model, and refers to the text provided to obtain the appropriate output.

[1121] "Real-time" refers to the responsiveness of data being processed immediately and provided to users the moment it is generated or transmitted.

[1122] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. The system of the present invention analyzes voice data and text data in real time and provides users with the necessary information.

[1123] Server configuration and processing

[1124] The server receives the voice data in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text API). This speech recognition tool uses the latest natural language processing technology and can convert voice data into text with high accuracy.

[1125] The server then uses a generative AI model (e.g., OpenAI GPT-3) to analyze the text data and summarize the key points of the meeting. By inputting prompts into this generative AI model, it is possible to extract the gist of the meeting discussion.

[1126] In addition, the server uses a sentiment analysis tool (e.g., IBM Watson Tone Analyzer) to analyze the atmosphere of the meeting and the emotions of the participants from the text data. The results of the sentiment analysis allow users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[1127] Finally, the server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[1128] Terminal configuration and handling

[1129] When a user joins a web conference late, the device captures the conference audio data in real time and sends the audio data in a stream format to the server. Chat messages generated during the conference are also sent to the server. The device receives the analysis results sent from the server in real time and displays them on the user interface.

[1130] User operations

[1131] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can join the discussion without delay by checking the information displayed on their device and making necessary comments or questions.

[1132] Specific example explanation

[1133] For example, suppose a user joins a regular sales meeting late at 10:15. The meeting started at 10:00, and the server has received the voice and chat data up to that point. When the user joins the meeting, the server sends a summary of the discussion so far and the results of sentiment analysis to the device. By checking this information, the user can immediately understand the progress of the meeting.

[1134] As another example, suppose a user joins a new product development meeting late at 2:20 PM. The meeting begins at 2:00 PM, and lively discussions are taking place about new effects. The server analyzes the voice and chat data to determine key points and participants' emotions. By checking the information displayed on the device, the user can quickly grasp the flow of the discussion and the opinions of the participants.

[1135] Example prompts for generative AI models

[1136] Here are some example prompts to input to a generative AI model:

[1137] Meeting text data:

[1138] 1. "Participant A: Regarding this project..."

[1139] 2. "Participant B: That's a great idea, but..."

[1140] 3. "Participant C: In response, I..."

[1141] Summarize the key points:

[1142] In this way, the system of the present invention allows even late-joining users to smoothly join the conference, thereby improving the productivity and efficiency of the conference.

[1143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1144] Step 1: Receiving and converting audio data

[1145] The server receives voice data sent from the device in real time. As input, it receives voice data from the device via WebSocket or RTMP protocol. This voice data is passed to a voice recognition tool (e.g., Google Cloud Speech-to-Text API) and converted into text data. Specifically, the server establishes a WebSocket connection and stores the voice data in a buffer. Once a certain amount of data has accumulated, it sends it to the voice recognition tool, which converts the voice data into text data. The converted text data is obtained as output.

[1146] Step 2: Summarize the discussion

[1147] The server inputs the converted speech-to-text data into a generative AI model (e.g., OpenAI GPT-3). As input, it cleans the converted text data and formats it into a format suitable for the generative AI model. It creates a prompt and supplies it to the generative AI model. Specifically, it formats the text data, inputs it into the generative AI model, and forms a prompt that generates a summary. As output, it obtains the summary data returned by the generative AI model.

[1148] Step 3: Sentiment analysis

[1149] The server passes the converted text data to a sentiment analysis tool (e.g., IBM Watson Tone Analyzer). As input, it sends the formatted text data to the sentiment analysis tool. Specifically, the server converts the text data into a format suitable for sentiment analysis and sends it to the sentiment analysis tool. It receives the results from the sentiment analysis tool and converts them into a format that is easy to analyze. As output, it obtains the sentiment analysis results.

[1150] Step 4: Generate and send integration results

[1151] The server integrates the summary information and the sentiment analysis results to form a single dataset. It receives the summary information and the sentiment analysis results as input. Specifically, the server integrates the summary information and the sentiment analysis results to generate a dataset in JSON format or similar. It then sends the integrated dataset to the device using WebSocket or HTTP POST. The integrated dataset is sent to the device as output.

[1152] Step 5: Capturing and Sending Data

[1153] The terminal captures the conference audio data in real time when a user joins a web conference late. As input, the terminal captures the conference audio data and sends it to the server in stream format. Specifically, the terminal starts audio capture when the user joins the conference application late. The terminal buffers a certain number of bytes of audio data and sends it to the server in stream format. As output, the captured audio data is sent to the server.

[1154] Step 6: Receive and display analysis results

[1155] The device receives the analysis results sent from the server in real time. As input, it monitors WebSocket connections and HTTP long polling to receive datasets from the server. Specific operations include analyzing the received datasets and displaying them in the user interface. The user can check the displayed information and chat or make comments as needed. As output, the user can check the progress of the meeting, summary information, and sentiment analysis results on the device.

[1156] By following these steps, even users who join late can quickly grasp the progress and emotional trends of the meeting and participate in the discussion efficiently.

[1157] (Application example 1)

[1158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1159] In conventional web conferencing systems, users who join a meeting late have difficulty quickly grasping the progress and atmosphere of the meeting and immediately and efficiently participating in the discussion. In addition, in order for users who join late to understand the overall picture and important points of the meeting, they have to ask other participants for explanations, which can reduce the productivity and efficiency of the meeting.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1161] In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis on the text data and providing the results, means for integrating the analysis results and providing them to users joining late, and means for displaying the summary information of the meeting and the sentiment analysis results on a smartphone or a head-mounted display, thereby enabling users joining late to quickly grasp the progress of the meeting, important points, and the emotions of the participants, and to participate in the discussion efficiently.

[1162] "Audio data" refers to sound information, including speech and dialogue, that occurs during a meeting, and is digital data that is sent to a server for analysis.

[1163] "Text data" is voice data converted into text, and is data in which the contents of the conference and the contents of statements are expressed as character information.

[1164] "Means for receiving voice data and converting it into text data" refers to the process, technology, or software for receiving voice data during a conference on a server and converting it into text information.

[1165] A "means for summarizing key points" is an algorithm or technology that extracts important information and the gist of discussions from what is said during a meeting and summarizes them concisely.

[1166] "Means for performing sentiment analysis of text data and providing the results" refers to technology or software that analyzes text data to understand the emotional state of the speaker and the atmosphere of the meeting, and provides that information to users.

[1167] "Means for integrating analytical results and providing them to late-joining users" refers to technologies and interfaces that compile summarized information and sentiment analysis results into a single, cohesive data set and provide it to late-joining users in real time.

[1168] A "smartphone" is a type of mobile phone, a portable device that has advanced computing power and connectivity and can run various applications.

[1169] A "head-mounted display (HMD)" is a display device worn by the user on the head to provide visual information, and is also used to display information in meetings.

[1170] This invention is a system that allows users who join a web conference late to quickly understand the progress and atmosphere of the conference and efficiently participate in the discussion. Below, we will explain the processing content and details of the program that realizes the system.

[1171] Server Processing

[1172] Receiving and converting audio data

[1173] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using a voice recognition tool. For voice recognition, a common voice recognition tool such as Google Speech Recognition API is used.

[1174] Summary of the discussion

[1175] The converted text data is then input into a generative AI model, which uses a Japanese-specific model such as "rinna / japanese-gpt-1b" to analyze the meeting text data and extract and summarize key points and the gist of the discussion.

[1176] sentiment analysis

[1177] The server performs sentiment analysis on the converted text data using sentiment analysis tools such as "daigo / bert-base-japanese-sentiment." This analysis allows the user to understand the atmosphere of the meeting and the emotions of the participants.

[1178] Generating and sending integration results

[1179] The server combines the summary information and the sentiment analysis results into a single dataset, which is then sent to the user's device in real time.

[1180] Terminal handling

[1181] Data capture and transmission

[1182] When a user joins a conference late, the user's device captures the conference's audio data in real time, and sends the audio data in a stream format to the server. At the same time, chat messages generated during the conference are also sent to the server.

[1183] Receiving and displaying analysis results

[1184] The analysis results sent from the server are instantly received on the user's device, which then displays the results in an interface that makes them easy to understand for the user, using a smartphone or head-mounted display (HMD).

[1185] User Actions

[1186] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, and the emotional trends of the participants. Users can check the information displayed on their devices and make necessary comments or questions.

[1187] Specific examples

[1188] Example 1: Meeting about new product displays

[1189] For example, if a staff member is late to a meeting about displaying a new product, this system can immediately grasp the key points of the meeting and the opinions of the participants, making it possible to take appropriate action.

[1190] Example prompts for generative AI models

[1191] "Summarize the following text. Generate a summary of a maximum of 50 characters and a minimum of 25 characters."

[1192] text:

[1193] We are starting discussions about a new product. We are currently discussing how to improve the product's packaging design. There is a strong consensus that a design based on the color red would be good.

[1194] This allows staff who join late to quickly catch up on information and participate effectively in the meeting.

[1195] In this way, the system of the present invention allows even late-joining users to join the conference efficiently, improving the productivity and efficiency of the conference.

[1196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1197] Step 1: Receiving and converting audio data

[1198] The server receives voice data sent from the user's device in real time. This voice data is converted into text data using the Google Speech Recognition API. The input is voice data, and the output is recognized text data. In this conversion process, the voice waveform data is analyzed and each phoneme is converted into the corresponding text information.

[1199] Step 2: Summarize the discussion

[1200] The server inputs the text data obtained in step 1 into a generative AI model. This generative AI model uses "rinna / japanese-gpt-1b" to analyze the text data, extract key points and the gist of the argument, and generate a summary. The input is text data, and the output is a summary that summarizes the main points. Specifically, the model analyzes language patterns, identifies important keywords and phrases, and generates a short summary.

[1201] Step 3: Sentiment analysis

[1202] The server performs sentiment analysis on the text data obtained in step 1. For this analysis, it uses "daigo / bert-base-japanese-sentiment." The input is the text data, and the output is the sentiment analysis result (positive, negative, neutral, etc.). Specifically, the model extracts keywords and phrases related to emotions in the text, calculates each sentiment score, and performs classification.

[1203] Step 4: Integrating the analysis results

[1204] The server integrates the summary information obtained in step 2 with the sentiment analysis results obtained in step 3. The input is the summary sentence and the sentiment analysis results, and the output is an integrated dataset. In this integration process, each analysis result is associated and formatted in a user-friendly format.

[1205] Step 5: Viewing the analysis results

[1206] The user device receives the integrated dataset sent from the server in real time and displays it on a smartphone or head-mounted display (HMD). The input is the integrated dataset, and the output is the information displayed on the user interface. Specifically, summary information and sentiment analysis results are displayed on the device display, providing an interactive interface that allows the user to understand immediately.

[1207] Step 6: User Actions

[1208] Users check the information displayed on their devices to understand the progress of the meeting and key points. By making specific comments and asking questions, users can catch up and participate in the discussion efficiently. The input is the displayed information of the analysis results, and the output is the user's actions. Specifically, users operate the interface, obtain the necessary information, and make comments.

[1209] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1210] This invention relates to a system that enables users who join a web conference late to quickly grasp the progress and atmosphere of the conference, enabling them to participate in the discussion efficiently.Furthermore, by combining it with an emotion engine that recognizes the user's emotions and provides appropriate feedback based on those emotions, more advanced support is realized.

[1211] Server Processing

[1212] Receiving and converting audio data

[1213] The server receives the voice data sent from the device in real time and converts it into text data using a voice recognition tool. The converted text data is temporarily stored.

[1214] Summary of the discussion

[1215] The converted text data is then fed into a generative AI model hosted on a server, which extracts and summarizes the key points and main points of the meeting. This summary information is then provided to the user.

[1216] sentiment analysis

[1217] The server then passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. This sentiment analysis allows users to understand not only the progress of the meeting but also the emotional reactions of the participants.

[1218] User emotion recognition

[1219] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[1220] Generating and sending integration results

[1221] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for immediate viewing by the user.

[1222] Terminal handling

[1223] Data capture and transmission

[1224] When a user joins a web conference late, the device captures the conference audio data in real time and transmits it to the server in a stream format. Chat messages that are generated are also transmitted to the server.

[1225] Receiving and displaying analysis results

[1226] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[1227] User Actions

[1228] Even if a user joins a web conference late, they can use this system to quickly grasp the progress of the conference, important points, the emotional trends of the participants, and feedback based on their own emotions, enabling them to participate in the discussion efficiently.

[1229] Specific examples

[1230] In the case of a regular meeting of the sales department

[1231] Suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins at 10:15, the server sends the summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[1232] New product development meeting

[1233] Suppose a user joins a new product development meeting late at 14:20. The meeting started at 14:00, and lively discussions were taking place about new production effects. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes users' emotions in real time and generates feedback. The user, who joined late at 14:20, can smoothly participate in the discussion by checking the information displayed on his / her device, understanding the flow of the discussion, participants' opinions, and feedback based on his / her own emotions.

[1234] In this way, the system of the present invention not only enables users who join a web conference late to quickly and efficiently grasp the situation of the conference and participate in the discussion, but also improves overall conference productivity and participant satisfaction by recognizing users' emotions and providing appropriate feedback.

[1235] The processing flow will be explained below.

[1236] Step 1:

[1237] The device verifies the user's authentication information to join the web conference late and connects to the conference. At this time, it begins capturing the conference's audio data in real time. It also captures the user's facial expression data and voice tone and sends them to the server as emotional data.

[1238] Step 2:

[1239] The device transmits the captured audio data to the server in a stream format, with minimal latency and in real time, along with chat messages and emotional data.

[1240] Step 3:

[1241] The server receives the voice data sent from the terminal and converts it into text data using a voice recognition tool. This conversion is performed in real time, and the converted text data is temporarily stored.

[1242] Step 4:

[1243] The server inputs the converted text data into a generative AI model to analyze the meeting content. This generative AI model extracts important points and the gist of the discussion and generates summary data. This summary data is temporarily stored.

[1244] Step 5:

[1245] The server passes the text data to a sentiment analysis tool to analyze the atmosphere of the meeting and the emotions of the participants. The results of the sentiment analysis are stored separately.

[1246] Step 6:

[1247] The server analyzes the user's emotional data sent from the device using an emotion engine. The emotion engine recognizes emotions in real time from the user's facial expressions and tone of voice. The recognized emotional data is reflected in the analysis results of the meeting.

[1248] Step 7:

[1249] The server combines the generated summary data, the results of the sentiment analysis, and the user sentiment data to generate a single, integrated dataset, which is then formatted as information for late-joining users.

[1250] Step 8:

[1251] The server then transmits the formatted, consolidated data set to the device, again in real time to minimize delays.

[1252] Step 9:

[1253] The terminal receives the integrated data set sent from the server, and the received data is displayed on the user interface for the user to easily check.

[1254] Step 10:

[1255] Users can view summary information and sentiment analysis results displayed on their devices, and also receive feedback based on their own emotional state, allowing them to quickly grasp the progress of the meeting, key points of discussion, participants' emotions, and their own state.

[1256] Step 11:

[1257] Based on the information they have checked, users can express their opinions and ask questions at the appropriate time, allowing them to participate smoothly in meetings. They can also refer to the displayed feedback and control their emotions while participating.

[1258] Example 2

[1259] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1260] In today's business environment, web conferencing is an important means of communication, but when users join late, it is difficult to grasp the situation and participate effectively in the discussion. Understanding the emotions of meeting participants is also essential, but there is a lack of ways to provide this information in real time. This makes it difficult for participants to smoothly join meetings without disrupting the flow of the discussion.

[1261] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing important points, means for performing sentiment analysis of the text data and providing the results, means for analyzing user sentiment and reflecting the data in the meeting analysis, and means for integrating the analysis results and providing them to users who join late. This makes it possible for users who join late to quickly and efficiently grasp the progress of the meeting and the sentiments of the participants, and actively participate in the discussion.

[1262] "Audio data" is data used to record or transmit sound in digital form.

[1263] "Text data" refers to data for recording linguistic information such as characters and symbols in digital form.

[1264] "Convert" refers to changing data of one format into data of another format.

[1265] "Summarizing" refers to extracting the key points of the original data and summarizing them in a short, concise form.

[1266] "Sentiment analysis" is a technology that analyzes emotional states from text data and audio data.

[1267] "Integrating" refers to bringing together multiple pieces of data or information into one coherent whole.

[1268] "User" refers to any individual or organization that uses the System.

[1269] A "means" refers to a device, method, or technique used to accomplish a particular purpose.

[1270] A "server" refers to a computer system that provides data and services to other computers (clients) on a computer network.

[1271] "Arriving late" means joining an event or activity (in this case a web conference) after the scheduled start time.

[1272] This invention is a system that allows users who join a web conference late to quickly grasp the progress and atmosphere of the conference and efficiently participate in the discussion. This system includes a series of processing steps: receiving and converting voice data, summarizing the discussion content, analyzing the sentiment of text data, recognizing users' emotions, and integrating and providing the analysis results.

[1273] The server receives the voice data from the device in real time and converts it into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[1274] The server then analyzes the text data using a generative AI model (e.g., OpenAI's GPT-3) to summarize the key points and main points of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants. This allows the system to understand not only the progress of the meeting but also the emotional reactions of the participants.

[1275] Furthermore, the server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. This engine includes technology that analyzes emotions from the user's voice and facial expressions. This emotion data is also reflected in the analysis results of the meeting.

[1276] The server combines the summary information, the results of the sentiment analysis, and the user's emotional data to generate a single, integrated dataset. This integrated dataset is sent to the device in real time and can be viewed immediately by the user. By viewing this information, the user can quickly grasp the progress and key points of the meeting, the emotional trends of participants, and feedback based on their own emotions.

[1277] As a concrete example, suppose a user joins a regular sales meeting late at 10:15. The server continues to receive audio and chat data from the meeting, which began at 10:00, and analyzes the user's emotional data in real time. When the user joins, the server sends a summary of the meeting, the results of the emotional analysis, and feedback based on the user's emotions to the device. The user can check this information, immediately understand the progress of the meeting, and participate with the appropriate emotions.

[1278] An example of a prompt is as follows:

[1279] "The regular sales meeting will start at 10:00. Please summarize the key points and the sentiment analysis results of the participants."

[1280] In the case of a new product development meeting, let's say a user joins late at 14:20. The meeting started at 14:00, and lively discussions about the new product were taking place. The server analyzes the voice and chat data to analyze key points and participants' emotions. It also analyzes the user's emotions in real time and generates feedback. When a user joins late, they can smoothly participate in the discussion by checking the information displayed on their device, understanding the flow of the discussion, participants' opinions, and feedback based on their own emotions.

[1281] In this way, by using the system of this invention, users who join a web conference late can not only quickly and efficiently grasp the situation of the conference and participate in the discussion, but also recognize the user's emotions and provide appropriate feedback.

[1282] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1283] Step 1:

[1284] Capture and transmit audio data

[1285] When a user joins a web conference late, the device captures the conference's audio data in real time through the microphone, and transmits the captured audio data in a stream format to the server. Chat messages generated during the conference are also transmitted from the device to the server.

[1286] Input and Output

[1287] Input: Audio data obtained from the device's microphone, chat messages during the meeting

[1288] Output: Streamed audio data and chat messages sent to the server.

[1289] Specific actions

[1290] When a user launches a web conferencing app on their device, the microphone is automatically enabled and audio data is captured. The audio data is converted into a streaming format using a dedicated library and sent to the server via the HTTPS protocol. Chat messages are also uploaded to the server in their original format.

[1291] Step 2:

[1292] Receiving voice data and converting it to text

[1293] The server receives the voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition tool (e.g., Google Cloud Speech-to-Text). The converted text data is temporarily stored on the server.

[1294] Input and Output

[1295] Input: Stream format audio data sent from the device

[1296] Output: Text data generated by the speech recognition tool

[1297] Specific actions

[1298] After receiving the voice data, the server converts it into text using the Google Cloud Speech-to-Text API, which is then stored in a database.

[1299] Step 3:

[1300] Discussion summary and sentiment analysis

[1301] The server inputs the converted text data into a generative AI model (e.g., OpenAI GPT-3) to generate a summary of the meeting. At the same time, it passes the text data to a sentiment analysis tool (e.g., IBM Watson) to analyze the atmosphere of the meeting and the emotions of the participants.

[1302] Input and Output

[1303] Input: Text data

[1304] Output: Summary information, sentiment analysis results

[1305] Specific actions

[1306] The server inputs the converted text data into a generative AI model via a Python script, generates summary information using prompt sentences, and passes the text data to a sentiment analysis tool to analyze participants' emotional states.

[1307] Step 4:

[1308] User emotion recognition

[1309] The server uses an emotion recognition engine (e.g., Microsoft Azure Face API) to recognize the user's emotions. It analyzes emotions based on image data acquired from the user's camera and audio data acquired from the microphone, and reflects this emotional data in the meeting analysis.

[1310] Input and Output

[1311] Input: User's facial expression data, voice data

[1312] Output: User emotion data

[1313] Specific actions

[1314] The server periodically sends image and audio data captured by the user's camera to the emotion recognition engine, which analyzes it and outputs the user's emotional state. This data is then integrated with other analysis results.

[1315] Step 5:

[1316] Generating and sending integration results

[1317] The server combines the summary information, the results of sentiment analysis, and the user's emotional data to generate a single, integrated dataset, which is then sent to the device in real time for the user to view immediately.

[1318] Input and Output

[1319] Input: Summary information, sentiment analysis results, user sentiment data

[1320] Output: Unified dataset

[1321] Specific actions

[1322] The server aggregates information from multiple data sources and generates a single dataset using a Python script, which is then sent to the terminal via HTTPS.

[1323] Step 6:

[1324] Receiving and displaying analysis results

[1325] The terminal instantly receives the integrated result data sent from the server and displays it in real time on the user interface (UI), allowing users to check the progress of the meeting, key points, participants' emotions, and feedback based on their own emotions.

[1326] Input and Output

[1327] Input: Integrated dataset sent from the server

[1328] Output: Analysis results displayed in the user interface

[1329] Specific actions

[1330] The device receives the integrated dataset sent via HTTPS and displays the analysis results on the screen using a built-in UI framework.

[1331] Step 7:

[1332] Viewing and Working with Users

[1333] Even if a user joins a web conference late, they can quickly grasp the progress of the conference, important points, participants' emotional trends, and feedback based on their own emotions by referring to the information displayed in the UI, allowing them to participate in the discussion efficiently.

[1334] Input and Output

[1335] Input: Analysis results that users see on the screen

[1336] Output: User understanding and reference for efficient discussion participation

[1337] Specific actions

[1338] Users can scroll and click to view the information displayed on the UI and obtain the necessary information to participate in the meeting appropriately.

[1339] (Application example 2)

[1340] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1341] In today's world, when users arrive late to a product information session in a physical store, it is difficult for them to quickly grasp the progress and important points. Furthermore, there is a lack of systems that provide users with emotional feedback and support appropriate purchasing decisions. Therefore, there is a need for a method to improve the shopping experience and participation efficiency of product information sessions.

[1342] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data and converting it into text data, means for analyzing the text data related to the meeting and summarizing the important points, means for performing sentiment analysis of the text data and providing the results, means for recognizing the user's emotions and generating feedback based on the emotions, and means for integrating the analysis results and providing them to users who join late. This enables users who join late in a physical store to quickly understand the situation and receive appropriate feedback.

[1343] "Audio Data" means sound signals recorded by a microphone or other sound capture device.

[1344] "Text data" refers to document data obtained by converting voice data into text.

[1345] "Analysis" refers to the act of analyzing audio data or text data for a specific purpose.

[1346] "Summarization" refers to extracting important points from text data and generating a shortened document.

[1347] "Sentiment analysis" is a technique for identifying emotional states based on text and audio data.

[1348] "Feedback" refers to information or advice provided to users based on analysis results and sentiment analysis results.

[1349] "User emotion" refers to the emotional state of the user that can be derived from their voice and facial expressions.

[1350] "Integration" refers to the act of combining various analytical results and data into a single dataset.

[1351] "Real-time" refers to the timing of processing data and providing results immediately.

[1352] A "generative AI model" refers to an algorithm that uses machine learning to generate new information from data.

[1353] A "prompt sentence" is text that is used as input for an AI model.

[1354] This invention relates to a system that enables users who arrive late to product information sessions or events at physical stores to quickly grasp the progress and important points, and participate efficiently. It also recognizes the user's emotions and provides appropriate feedback based on those emotions.

[1355] Server Processing

[1356] The server receives the voice data in real time and converts it into text data using a speech recognition tool. Specifically, one available speech recognition tool is the speech_recognition library. The converted text data is temporarily saved. This text data is then input into a generative AI model, which summarizes the key points of the meeting and the main points of the discussion. This summary information is also later provided to the user. The generative AI model uses, for example, the Hugging Face transformers library.

[1357] Sentiment analysis is performed by passing text data to a sentiment analysis tool, such as the company's proprietary EmotionRecognizer module, which analyzes the atmosphere of the meeting and the emotions of the participants.

[1358] Furthermore, the server uses an emotion engine to recognize the user's emotions, including analyzing emotions from the user's voice and facial expressions. This emotion data is reflected in the analysis results of the meeting.

[1359] Finally, the server combines the summary information, the sentiment analysis results, and the user's sentiment data to generate a single integrated dataset, which is sent to the device in real time for immediate user review.

[1360] Terminal handling

[1361] The device captures the audio data of a product presentation in a physical store in real time when the user joins late and sends it to the server in a stream format. Specific use cases include smartphones and smart glasses. Chat messages during the event are also sent to the server.

[1362] The analysis results sent from the server are instantly received on the device and displayed on the user interface, along with feedback based on the user's emotions.

[1363] Specific examples

[1364] Consider the example of a user arriving late to a new product briefing. Suppose the user joins 20 minutes after the briefing has started. The server receives and analyzes voice and chat data in real time from the start of the briefing. When the user joins, the server sends a summary of the briefing, sentiment analysis results, and feedback based on the user's sentiment to the device. The user can check this information, immediately understand the progress of the briefing, and receive appropriate feedback, allowing them to smoothly join the briefing.

[1365] Generative AI model prompt:

[1366] Please extract and summarize the following information from the audio data.

[1367] 1. Summary of the meeting

[1368] 2. Main opinions of participants

[1369] 3. Emotional atmosphere

[1370] This means that even if users arrive late, they can quickly obtain the information they need and participate in the product briefing efficiently.

[1371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1372] Step 1:

[1373] When a user joins a product briefing session at a physical store late, the device captures audio data of the session in real time. The captured audio data is sent to the server in stream format. In this step, audio data is recorded in real time using the device's microphone and sent to the server via the network. The input is the captured audio data, and the output is the audio data sent to the server.

[1374] Step 2:

[1375] The server converts the received voice data into text data in real time using a voice recognition tool. Specifically, it converts the voice data into text using the speech_recognition library. In this step, the input is voice data and the output is text data. The server temporarily stores the converted text data.

[1376] Step 3:

[1377] The server analyzes the converted text data using a generative AI model and summarizes the key points. In this step, the text data is input to the summarization model using Hugging Face's transformers library to obtain summarized text data. The input is text data, and the output is summarized text.

[1378] Step 4:

[1379] The server passes the summarized text data to the sentiment analysis tool to perform sentiment analysis on the data. Specifically, it uses the EmotionRecognizer module to recognize emotions in the text data and identify emotional states. In this step, the input is the summarized text and the output is the sentiment analysis results.

[1380] Step 5:

[1381] The server analyzes the user's voice and facial expressions to recognize the user's emotions. This involves analyzing voice data and image data using facial recognition technology. The server determines the user's emotional state based on this data. The input is the user's voice data and facial image data, and the output is the user's emotional state.

[1382] Step 6:

[1383] The server integrates the summary information, sentiment analysis results, and user emotion data to generate a single integrated dataset, which contains important information and emotional feedback that users need to understand. The inputs are the summary information, sentiment analysis results, and user emotion data, and the output is the integrated dataset.

[1384] Step 7:

[1385] The server transmits this integrated dataset to the terminal in real time. The transmitted data is immediately received by the terminal and displayed on the user interface. The input is the integrated dataset, and the output is the analysis results and feedback displayed on the terminal.

[1386] Step 8:

[1387] By checking the analysis results and feedback displayed on the device, users can grasp the progress of the product briefing, important points, and feedback based on their own emotions, allowing them to participate in the event efficiently. The input is the analysis results and feedback displayed on the device, and the output is improved user understanding and participation efficiency.

[1388] By following the above steps, users who arrive late at a store can quickly understand the situation and participate effectively.

[1389] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1391] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1392] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1393] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1394] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1395] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1396] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1397] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1398] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1399] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1400] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1401] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1402] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1403] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1404] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1405] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1406] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1407] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1408] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1409] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1410] The following is further disclosed regarding the above embodiment.

[1411] (Claim 1)

[1412] means for receiving voice data and converting it into text data;

[1413] A means of analyzing text data about meetings and summarizing key points;

[1414] means for performing sentiment analysis on text data and providing the results;

[1415] A means to consolidate and provide the results of the analysis to late participants;

[1416] A system including:

[1417] (Claim 2)

[1418] 10. The system of claim 1, wherein the system analyzes the voice data and the text data in real time.

[1419] (Claim 3)

[1420] 10. The system according to claim 1, further comprising a sentiment analysis means for analyzing the sentiment trends of the conference participants.

[1421] "Example 1"

[1422] (Claim 1)

[1423] means for receiving voice data and converting it into text data;

[1424] A means of analyzing text data about meetings and summarizing key points;

[1425] means for performing sentiment analysis on text data and providing the results;

[1426] A means to consolidate and provide the results of the analysis to late participants;

[1427] a means for summarizing text data using a generative AI model;

[1428] a means of processing and providing data in real time;

[1429] A system including:

[1430] (Claim 2)

[1431] 10. The system of claim 1, wherein the system analyzes the audio data and text data in real time and integrates the key points of the meeting with the sentiment analysis results.

[1432] (Claim 3)

[1433] 10. The system of claim 1, further comprising means for inputting a prompt sentence to the generative AI model.

[1434] "Application Example 1"

[1435] (Claim 1)

[1436] means for receiving voice data and converting it into text data;

[1437] A means of analyzing text data about meetings and summarizing key points;

[1438] means for performing sentiment analysis on text data and providing the results;

[1439] A means to consolidate and provide the results of the analysis to late participants;

[1440] A means to display summary information of the meeting and the results of sentiment analysis on a smartphone or head-mounted display,

[1441] A system including:

[1442] (Claim 2)

[1443] 10. The system of claim 1, wherein the system analyzes the voice data and the text data in real time.

[1444] (Claim 3)

[1445] 10. The system according to claim 1, further comprising a sentiment analysis means for analyzing the sentiment trends of the conference participants.

[1446] "Example 2: Combining Emotion Engines"

[1447] (Claim 1)

[1448] means for receiving voice data and converting it into text data;

[1449] A means of analyzing text data about meetings and summarizing key points;

[1450] means for performing sentiment analysis on text data and providing the results;

[1451] A means of analyzing user emotions and reflecting that data in meeting analysis,

[1452] A means to consolidate and provide the results of the analysis to late participants;

[1453] A system including:

[1454] (Claim 2)

[1455] 10. The system of claim 1, wherein the system analyzes the voice data and the text data in real time.

[1456] (Claim 3)

[1457] 10. The system according to claim 1, further comprising a sentiment analysis means for analyzing the sentiment trends of the conference participants.

[1458] "Application example 2 when combining emotion engines"

[1459] (Claim 1)

[1460] means for receiving voice data and converting it into text data;

[1461] A means of analyzing text data about meetings and summarizing key points;

[1462] means for performing sentiment analysis on text data and providing the results;

[1463] a means for recognizing a user's emotions and generating emotion-based feedback;

[1464] A means to consolidate and provide the results of the analysis to late participants;

[1465] A system including:

[1466] (Claim 2)

[1467] 10. The system of claim 1, wherein the system analyzes the voice data and the text data in real time.

[1468] (Claim 3)

[1469] 10. The system of claim 1, further comprising a sentiment analysis means for analyzing the sentiment trends of conference participants and users. [Explanation of symbols]

[1470] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving voice data and converting it into text data; A means of analyzing text data about meetings and summarizing key points; means for performing sentiment analysis on text data and providing the results; A means to consolidate and provide the results of the analysis to late participants; A system including:

2. 10. The system of claim 1, wherein the system analyzes the audio data and the text data in real time.

3. 2. The system according to claim 1, further comprising emotion analysis means for analyzing the emotional trends of the conference participants.

Images