System

Abstract

To provide a system capable of improving the quality of a conference and being useful for improving the conference skill of participants.SOLUTION: The conference system includes means for receiving a video stream and an audio stream during a conference, means for analyzing the received video stream and audio stream and analyzing speech contents, expressions, and movements of participants in real time, means for detecting silence based on an analysis result and specifying a participant to be urged to speak, means for transmitting a notification for urging to speak, means for summarizing an outline of the conference at regular time intervals based on the speech contents and a progress status of a discussion during the conference, means for transmitting the summarized outline of the conference to terminals of all participants, means for automatically generating minutes based on all speeches and analysis data after the conference ends, means for performing scoring based on the number of speeches and speech contents of each participant, and means for individually transmitting a result scored based on the number of speeches and speech contents.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Silence and stalled discussions are major factors that reduce productivity in meetings. Furthermore, if a particular participant is not concentrating or appears sleepy during the meeting, it has a negative impact on the overall progress of the discussion. It is also difficult to grasp and share the content of comments and key points of discussions in real time, which hinders efficient meeting management. A system that solves these problems and enables highly productive meeting management is needed. [Means for solving the problem]

[0005] The present invention relates to a system that receives video and audio streams during a conference and analyzes them in real time. Specifically, the system includes a means for analyzing the received video and audio streams and analyzing the participants' comments, facial expressions, and movements. It also includes a means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak. This makes it possible to automatically encourage appropriate participants to speak when the conference is stalled. The system also provides a means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion, and transmits the summarized conference summaries to all participants' terminals. After the conference ends, the system also includes a means for automatically generating minutes based on all comments and analysis data, and for scoring each participant based on the number of comments and the content of their comments. This provides a system that can improve the quality of conferences and help participants improve their conference skills.

[0006] A "video stream" is a data format for transmitting and receiving video data of conference participants in real time.

[0007] An "audio stream" is a data format for transmitting and receiving audio data of conference participants in real time.

[0008] "Analyzing" means analyzing the participants' statements, facial expressions, and movements based on information obtained from the video stream and audio stream, and grasping a specific situation.

[0009] "Content of remarks" refers to the specific words and content spoken by meeting participants during the meeting.

[0010] "Facial expressions" refer to facial expressions and movements that show emotions of conference participants.

[0011] "Movement" refers to the physical movements and changes in posture of meeting participants.

[0012] "Silence" refers to a period of time when no meeting participants have spoken.

[0013] "Encouraging someone to speak" means taking action to ask a specific participant to speak in order to break the silence.

[0014] "Identifying" refers to identifying specific participants or situations for optimal action based on the analysis results.

[0015] "Sending a notification" means sending a digital message to convey specific information to the terminals of the conference participants.

[0016] "Summarizing the meeting outline" means organizing the main discussions and important points of the meeting in a form that can be understood in a short amount of time.

[0017] "Minutes" are a written record of what was discussed and what decisions were made during a meeting.

[0018] "Scoring" involves setting evaluation indicators based on the number of comments and content of each participant's comments, and then evaluating them numerically. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The system of the present invention is designed to analyze the video and audio streams of a conference in real time and facilitate the progress of the conference. A specific embodiment of this system will be described below.

[0041] The server receives the video and audio streams of the conference. It then analyzes the received video and audio streams to analyze the participants' speech, facial expressions, and movements in real time. Based on the analysis results, the server detects silence and identifies participants who should be encouraged to speak. It also sends notifications to encourage them to speak.

[0042] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The summarized meeting summary is sent to the devices of all participants. After the meeting ends, minutes are automatically generated based on all comments and analysis data. Each participant is scored based on the number of comments and the content of their comments, and the results are sent individually.

[0043] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[0044] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0045] The operation of the system will be explained below using specific examples.

[0046] Example 1: Silence detection and prompting

[0047] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be prompted to speak next. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who then gives their opinion based on it.

[0048] Example 2: Meeting summary and sharing

[0049] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[0050] Example 3: Minutes and scoring

[0051] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[0052] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[0056] Step 2:

[0057] The server analyzes the received video and audio streams. Specifically, it uses a generative AI model to analyze participants' speech, facial expressions, and movements in real time. Speech is converted into text, and facial expressions and movements are digitized as analysis results.

[0058] Step 3:

[0059] The server detects silence during the conference based on the analysis results. If no speech is made for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings.

[0060] Step 4:

[0061] When the server detects silence, it prompts the most appropriate participant to speak. It identifies participants who are not concentrating or appear sleepy from the analysis results and sends them a notification to prompt them to speak.

[0062] Step 5:

[0063] The device displays the notification received from the server to the user. Specifically, it displays a message such as "Please state your opinion next" on the screen to encourage the user to speak.

[0064] Step 6:

[0065] The user checks the notification displayed on the device and follows the instructions to express their opinions or ask questions. The comments are then sent back to the server via the device.

[0066] Step 7:

[0067] The server summarizes the meeting at regular intervals (for example, 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. This summary is generated in text format.

[0068] Step 8:

[0069] The server sends a summary of the meeting to all participants' terminals, which receive the summary message and display it to the user.

[0070] Step 9:

[0071] As the meeting progresses, the server continuously records what is said and the analytical data, so that all comments and discussions are stored in a log.

[0072] Step 10:

[0073] After the meeting, the server automatically generates minutes based on the accumulated remarks and analysis data. Specifically, it organizes the remarks made by each speaker and important points of discussion.

[0074] Step 11:

[0075] The server scores each participant based on the number and content of their comments. For example, the evaluation score is calculated based on multiple indicators such as the number of comments, the quality of the comments, and the degree of contribution to the discussion.

[0076] Step 12:

[0077] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[0078] Step 13:

[0079] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement.

[0080] The above are the specific processing steps of the present system based on the claims. This system allows meetings to proceed smoothly and enables highly productive discussions.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] Conventional conference systems lack sufficient support functions to facilitate smooth meeting progress, making it difficult for all participants to participate effectively in discussions. Furthermore, recording comments, generating minutes, and analyzing the content of comments must be done manually, which requires a great deal of time and effort. Another issue is that there are few ways to grasp participants' own contributions to the discussion, making it difficult to obtain specific feedback to improve the quality and efficiency of the meeting.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes means for receiving video data and audio data during a conference, means for analyzing the received video data and audio data and analyzing the participants' comments, facial expressions, and movements in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion, means for transmitting the summarized conference summaries to all participants' terminals, means for automatically generating minutes after the conference based on all comments and the analysis data, means for scoring each participant based on the number of comments and the content of their comments, means for individually transmitting the scoring results based on the number of comments and the content of their comments, means for using computer vision technology to analyze the video data, means for using voice recognition technology to analyze the audio data, means for monitoring the audio level to detect silence for a certain period of time, means for notifying all participants of the generated conference summary, and means for distributing the minutes and scoring results to all participants.This allows the conference to proceed smoothly and allows all participants to participate efficiently in the discussion. In addition, automated recording and analysis of meetings reduces effort and time, and allows each participant to understand their own contribution and receive specific feedback that can help improve their meeting skills.

[0086] "Video data during a meeting" refers to visual information acquired through a camera while the meeting is in progress, and includes visual content such as the movements and facial expressions of participants.

[0087] "Audio data" refers to acoustic information acquired through a microphone during a conference, including speeches made by participants and other audio information.

[0088] "Means for receiving" refers to devices or software that provide the function of acquiring video data and audio data and inputting them into the server.

[0089] "Means for analyzing" refers to an algorithm or computer program that processes the received data and extracts specific information.

[0090] "Real-time analysis" refers to the act of processing acquired data immediately to obtain results, providing information immediately without delay.

[0091] "Silence detection means" means a device or software that provides the functionality to identify and detect periods of silence within audio data.

[0092] "Means for identifying participants who should be encouraged to speak" refers to algorithms or computer programs that use the analyzed data to identify the next person to speak.

[0093] "Means for sending notifications" refers to machines and programs for sending messages or alerts to specific participants.

[0094] A "meeting summary tool" refers to an algorithm or computer program that concisely summarizes what was discussed during a meeting.

[0095] "Means for sending a conference summary to all participants' terminals" refers to machines and programs that distribute messages to convey summarized information to all participants.

[0096] "Means for automatically generating minutes" refers to algorithms or computer programs that automatically create detailed records of meetings based on what was said during the meeting and analytical data.

[0097] "Means for scoring" refers to an algorithm or computer program that evaluates the number and content of each participant's comments and calculates a score based on that.

[0098] "Means for individual transmission" refers to a machine and program that provides the functionality for transmitting scoring results to each participant's terminal in a specific format.

[0099] "Computer vision technology" refers to a collection of techniques and algorithms used to extract information from digital images and video.

[0100] "Speech recognition technology" refers to a collection of techniques and algorithms for analyzing voice data and converting it into text.

[0101] "Means for monitoring audio levels" refers to an algorithm or computer program that continuously monitors the volume of audio data and detects fluctuations in the volume.

[0102] The system of the present invention is designed to analyze the video and audio streams of a conference in real time to facilitate the progress of the conference. A specific embodiment of this system will be described below, with each role played by a server, a terminal, and a user.

[0103] Server Roles

[0104] The server processes data and provides functions to support the conference through the following procedures:

[0105] 1. Receiving data:

[0106] The server receives video and audio data during the conference using a communication protocol such as WebRTC, which transmits real-time video and audio streams from each participant to the server.

[0107] 2. Video and audio data analysis:

[0108] The received video data is analyzed using OpenCV and other computer vision technologies, specifically using facial recognition and expression analysis algorithms to understand the participants' facial expressions and movements.

[0109] The voice data is converted into text using voice recognition technology (for example, Google Cloud Speech-to-Text API), which makes the spoken content available as text data.

[0110] 3. Silence detection and prompting:

[0111] The server continuously monitors the audio level and detects silence for a certain period (for example, 10 seconds) using a specific algorithm that monitors the audio level.

[0112] When silence is detected, an algorithm determines who should be the next person to speak and sends them a notification, such as "You're next."

[0113] 4. Summary of the meeting:

[0114] Based on what was said during the meeting and the progress of the discussion, a generative AI model is used to summarize the main topics and decisions of the meeting at regular intervals (for example, every 15 minutes).

[0115] A summary of the meeting is sent to all participants' devices, making it easier for them to understand the flow of the meeting.

[0116] 5. Generate transcripts:

[0117] After the meeting, minutes are automatically generated based on all statements and analysis data. The generated minutes include a detailed record of what each speaker said.

[0118] 6. Scoring:

[0119] Each participant is scored based on the number of comments and the content of their comments. This scoring is based on the evaluation criteria of the frequency, importance, and contribution of comments.

[0120] The scored results are sent individually to each participant's device.

[0121] Device Role

[0122] The terminal provides the user with visual and auditory information about the data sent from the server, and transmits user operations to the server.

[0123] 1. Displaying data:

[0124] The terminal displays the video and audio streams sent from the server, allowing users to view the contents of the conference in real time.

[0125] 2. Capturing what users say and do:

[0126] When a user speaks, the device captures the user's voice and sends it to the server. It also captures the user's facial expressions and movements and sends this data to the server for analysis.

[0127] 3. Notification and Summary Display:

[0128] When a notification encouraging speaking or a summary of the conference is sent from the server, the terminal displays this to the user as a pop-up notification or message.

[0129] User Roles

[0130] Users can participate in the conference and make comments and hold discussions as usual. In addition, they can check notifications and summaries from the server and use them to express their opinions and advance the discussion.

[0131] 1. Statement:

[0132] The user speaks through a microphone, and the content of the speech is transmitted to the server via the terminal.

[0133] Facial expressions and movements are also captured by the camera and sent to the server.

[0134] 2. Responding to Notices:

[0135] When a user receives a notification from the server prompting the user to speak, the user responds by speaking.

[0136] 3. Review of minutes and scoring results:

[0137] After the meeting, users can check the minutes and scoring results displayed on their device to help improve their own meeting skills.

[0138] Specific examples

[0139] Example 1: Silence detection and prompting

[0140] During a meeting, if a participant finishes speaking and there is a 10-second silence, the server detects the silence and identifies the next participant to be prompted to speak. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who can then express their opinion.

[0141] Example 2: Meeting Summary

[0142] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[0143] Example 3: Minutes and scoring

[0144] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[0145] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] Receiving data

[0149] The server receives video and audio data during the conference using the WebRTC protocol.

[0150] Input: Real-time video and audio streams sent from each participant's device.

[0151] Data processing: Video and audio data are received in the form of packets.

[0152] Output: Received video and audio data.

[0153] How it works: The server uses the WebRTC protocol to receive video and audio streams from each participant in real time, store them in a database, and queue them for processing.

[0154] Step 2:

[0155] Video data analysis

[0156] The server analyzes the received video data using the OpenCV library.

[0157] Input: Video data received via the WebRTC protocol.

[0158] Data processing: Applying facial recognition and expression analysis algorithms to extract specific facial features and changes in facial expressions.

[0159] Output: Participant's facial expression and movement data.

[0160] Specific operation: The server uses OpenCV to detect facial features for each video frame and analyzes changes in facial expression and eye movements, thereby classifying facial expressions into categories such as "smiling," "serious," and "sleepy."

[0161] Step 3:

[0162] Analysis of audio data

[0163] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0164] Input: Audio data received via the WebRTC protocol.

[0165] Data processing: Using voice recognition technology, voice data is converted into text data.

[0166] Output: Text data of the speech.

[0167] What it does: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts what is said into text in real time, which is then stored in a database for further analysis and recording.

[0168] Step 4:

[0169] Silence Detection

[0170] The server monitors the analyzed audio data and detects periods of silence.

[0171] Input: Speech and speech level data converted to text.

[0172] Data processing: Based on the voice level data, monitor whether there is any continuous speech for a specific period (for example, 10 seconds).

[0173] Output: Silence detection results.

[0174] What happens: The server checks that the audio level is below a certain threshold and then runs an algorithm that detects silence if no speech is heard for 10 consecutive seconds.

[0175] Step 5:

[0176] Sending a call to speak

[0177] If a period of silence continues, the server identifies the participant who should next be prompted to speak and sends a notification.

[0178] Input: Silence detection results and analyzed facial expression data.

[0179] Data processing: Based on the analysis results obtained from video and audio data, participants who should be encouraged to speak are identified.

[0180] Output: Informational message.

[0181] Specific operation: The server sends a notification to the identified participant's device saying, "Please give your opinion next." This notification is displayed as a pop-up message on the device.

[0182] Step 6:

[0183] Conference Summary

[0184] The server periodically summarizes the contents of the conference.

[0185] Input: Text data converted from audio data and discussion progress.

[0186] Data processing: Using NLP techniques, we analyze what is being said and extract key topics and decisions.

[0187] Output: Meeting summary text.

[0188] Specific operation: The server uses a generative AI model to summarize the contents of the meeting every 15 minutes and sends the summary text to the devices of all participants.

[0189] Step 7:

[0190] Generate meeting minutes

[0191] After the meeting ends, the server automatically generates minutes based on all statements and analysis data.

[0192] Input: Text data and analysis data of all statements.

[0193] Data processing: Integrate the statements and analytical data and organize them into detailed records.

[0194] Output: Auto-generated meeting transcript.

[0195] Specific operation: The server automatically generates minutes based on all data collected during the meeting and sends them to each participant's device. The minutes include the remarks and decisions made by each participant.

[0196] Step 8:

[0197] Sending scoring results

[0198] The server scores each participant based on the number of comments and the content of their comments, and sends the results individually.

[0199] Input: All speech content and number of speeches data.

[0200] Data processing: A score is calculated for each speaker based on the number of comments, importance, and contribution of each speaker.

[0201] Output: Scoring results.

[0202] How it works: The server evaluates and scores each participant's performance, taking into account the frequency of their comments and the importance of their content. The results are sent to each participant's device as individual notifications.

[0203] These are the specific processing steps of the system. This allows meetings to proceed smoothly and enables all participants to participate in the discussion efficiently. In addition, the automatic generation of meeting minutes and feedback of scoring results can contribute to improving the quality of meetings and the skills of participants.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Conventional conference systems have difficulty efficiently and effectively managing tasks such as encouraging participants to speak and creating meeting minutes. In particular, current conference management systems are insufficient in production sites such as factories, where rapid decision-making and communication are required. Furthermore, there is no system for analyzing participants' facial expressions and movements in real time, making it difficult to properly grasp the status of participants during the meeting, which can slow down the progress of the meeting. Therefore, an effective system to solve these issues is needed.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for receiving video and audio streams during a conference, means for analyzing the received video and audio streams and analyzing the speech content, facial expressions, and movements of participants in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for analyzing the video and audio streams and encouraging specific participants to speak to support in-factory conferences, and means for applying a facial expression analysis model using the video and audio streams, identifying facial expressions based on the analysis results, and determining participants who should be encouraged to speak, thereby enabling quick decision-making and efficient communication within the factory.

[0209] A "video stream" is a series of video data captured during a conference via a video input device such as a camera.

[0210] An "audio stream" is a series of audio data captured during a conference via an audio input device such as a microphone.

[0211] "Real-time analysis" refers to the process of analyzing captured video and audio streams in real time and providing results without delay.

[0212] "Silence detection" refers to determining whether a silence state has occurred for a certain period of time or longer based on the analyzed audio stream.

[0213] "Speaker prompt notification" is a function that sends a message to specific participants to prompt them to speak when silence is detected.

[0214] The "facial expression analysis model" is a machine learning model for analyzing participants' facial expressions obtained from video streams and estimating their emotional states.

[0215] The "analysis results" are information on speech content, facial expressions, and movements obtained from the video stream and audio stream data.

[0216] "Scoring" is the process of evaluating each participant based on the number and content of their comments.

[0217] Minutes are documents that record what was said and what was decided at a meeting.

[0218] "Factory meetings" are meetings held in factories for the purpose of planning and solving problems related to production.

[0219] "Notification means" is a function for sending messages and instructions from the system to participants.

[0220] "Multimodal analysis" is the process of simultaneously analyzing multiple different data sources (such as video and audio streams) and utilizing the results in a comprehensive manner.

[0221] The present invention provides a system for smoothly conducting meetings in a factory and promoting efficient decision-making. Specific embodiments for implementing this system will be described below.

[0222] System Overview

[0223] This system consists of three roles: a server, a terminal, and a user. The server plays a central role in receiving and analyzing the video and audio streams of the conference. The terminal is a device through which participants interact with the system, displaying the video and audio streams and receiving notifications. The user participates in the conference, expressing their opinions and participating in discussions.

[0224] Hardware and Software Configuration

[0225] Hardware: Camera, microphone, display, smartphone.

[0226] Software: Python, OpenCV (video stream processing), TensorFlow (facial expression analysis model), Numpy (data processing).

[0227] Data processing and calculation

[0228] Server Processing

[0229] 1. Receiving video and audio streams

[0230] The server receives the video and audio streams from the camera and microphone in real time.

[0231] 2. Real-time analysis

[0232] We use OpenCV to detect participants' faces from the video stream, TensorFlow to analyze detected facial expressions and estimate participants' emotional states, and Numpy to analyze the volume levels of the audio stream and detect silence.

[0233] 3. Detect silence and send notifications

[0234] If silence continues for a certain period of time, the system will identify the participant who should be prompted to speak next based on the analysis results and send a notification to that participant, which will be displayed on their device.

[0235] 4. Meeting summary and minutes generation

[0236] Based on the main comments made during the meeting and the progress of the discussion, a summary of the meeting is generated at regular intervals (for example, every 15 minutes) and sent to the devices of all participants. After the meeting, minutes are automatically generated based on all comments and analysis data, and the number and content of comments made by each participant are scored.

[0237] Terminal handling

[0238] 1. Displaying video and audio streams

[0239] The terminal displays the video and audio streams sent from the server, allowing participants to visually understand the contents of the conference.

[0240] 2. Receiving and Displaying Notifications

[0241] It receives notifications from the server and displays them to users, who can then express their opinions and advance the discussion.

[0242] Specific examples

[0243] If there is silence during a meeting, the system will:

[0244] 1. At the beginning of the meeting, the statement is made, "We will begin discussing the design of the new product."

[0245] 2. If there is a period of silence, the server detects this and identifies the next participant to speak.

[0246] 3. A notification will appear on the device of the person you are speaking to saying, "Yamada-san, please comment on the next steps."

[0247] 4. This allows Yamada to express his opinion and the meeting proceeds smoothly.

[0248] An example prompt is:

[0249] Meeting start: "We're going to start discussing the design of our new product."

[0250] There is silence for a while.

[0251] Prompt to speak: "Yamada-san, please comment on next steps."

[0252] In this way, this system enables smooth progress in factory meetings and efficient decision-making.

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] Receiving video and audio streams

[0256] The server receives video and audio streams in real time through cameras and microphones. The input is the video and audio data of the meeting, and the output is the raw data stored and prepared for analysis. In this step, the server continuously acquires data by capturing the streams from each device.

[0257] Step 2:

[0258] Real-time analysis of video streams

[0259] The server uses OpenCV to detect participants' faces from the video stream. The input is the received video stream data, and the output is the location information of the detected faces. This face location information becomes the input for subsequent facial expression analysis. Specifically, the server recognizes the contours of the face from the video frame and identifies its location.

[0260] Step 3:

[0261] Application of facial expression analysis model

[0262] The server performs facial expression analysis on the detected faces using TensorFlow. The input is image data of salient faces and a pre-trained facial expression analysis model, and the output is an estimated emotional state (e.g., joy, anger, sadness, happiness, etc.). This is the process of applying the facial expression analysis model to classify facial expressions.

[0263] Step 4:

[0264] Parsing an Audio Stream

[0265] The server uses Numpy to analyze the volume level of the audio stream and detect silence. The input is the audio stream data acquired in real time, and the output is the volume level (silence or not) judgment result. Specifically, it calculates the amplitude of the audio data and determines silence if it is below a preset threshold.

[0266] Step 5:

[0267] Detect silence and send prompts

[0268] If silence continues for a certain period of time (e.g., 10 seconds), the server identifies the participant who should be prompted to speak next based on the results of facial expression analysis. The input is the silence detection result and the results of facial expression analysis, and the output is the identified participant ID and the corresponding prompt to speak. The server then sends a "please speak next" notification to this identified participant.

[0269] Step 6:

[0270] Viewing video and audio streams

[0271] The terminal receives the video and audio streams sent from the server and displays them to the user. The input is the video and audio stream data from the server, and the output is the display of that data on the terminal. This allows the user to grasp the status of the conference in real time.

[0272] Step 7:

[0273] Receiving and Viewing Notifications

[0274] The terminal receives a prompt to speak from the server and displays it to the user. The input is the prompt to speak notification data from the server, and the output is the display of the notification to the user. Upon receiving this notification, the user can express their opinion or advance the discussion.

[0275] Step 8:

[0276] Generate meeting summaries and minutes

[0277] The server generates a summary of the meeting at regular intervals (e.g., every 15 minutes) based on the main comments made during the meeting and the progress of the discussion, and sends it to the devices of all participants. After the meeting ends, it automatically generates minutes based on all comments and analysis data, scoring the number and content of comments made by each participant. The input is all comment data and analysis results from the meeting, and the output is summary information and minutes. Specifically, it uses natural language processing technology to create summaries, extract necessary information, and document it.

[0278] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0279] The present invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. Specific embodiments of this system are described below.

[0280] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows participants' speech, facial expressions, actions, and emotions to be analyzed in real time. Speech is converted into text, and facial expressions and actions are converted into data as analysis results. The emotion engine also recognizes participants' emotional states from their facial expressions and tone of voice.

[0281] Based on the analysis results, the server detects silence during the meeting. If no one speaks for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings. When silence is detected, the server also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. For example, it sends a notification encouraging participants who are determined to be not concentrating or whose emotional state is changing negatively to speak.

[0282] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[0283] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0284] The operation of the system will be explained below using specific examples.

[0285] Example 1: Silence detection and prompting

[0286] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be encouraged to speak next. If the emotion engine's analysis determines that a particular participant is losing concentration or showing negative emotions, it sends that participant a notification saying, "Please speak next." The device displays this notification to the user, who then expresses their opinion based on it.

[0287] Example 2: Meeting summary and sharing

[0288] As the meeting progresses, the server summarizes the content of the discussion every 15 minutes. For example, it briefly summarizes the main topics and decisions discussed in the first 15 minutes and sends this content to the devices of all participants. The analysis results of the emotion engine are also taken into account, and summary information corresponding to increases in negative emotions and decreases in positive emotions is provided. The device displays this summary information to the user, allowing them to always understand the progress of the meeting.

[0289] Example 3: Minutes and scoring

[0290] After the meeting ends, the server automatically generates minutes based on the accumulated content of comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding a new project proposal" and "Speaker B: Opinions on the proposal." The analysis results of the emotion engine are also reflected, and the emotional state of each speaker is included in the minutes. In addition, each speaker is scored based on the number of comments and content of their comments. For example, they might be evaluated as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments." The scoring also takes into account the quality of the comments and the emotional state. These results are sent individually to each participant's device, allowing users to check their own performance.

[0291] The above is a specific embodiment of the present invention. This system not only allows meetings to proceed smoothly, but also allows for more effective discussions by taking into account the emotional states of participants. The introduction of an emotion engine will improve the quality of meetings and also help improve individual meeting skills.

[0292] The processing flow will be explained below.

[0293] Step 1:

[0294] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[0295] Step 2:

[0296] The server uses generative AI models and emotion engines to analyze the incoming video and audio streams in real time, converting participants' speech into text and digitizing their facial expressions, movements, and emotional states.

[0297] Step 3:

[0298] Based on the analysis results, the server monitors speech content, facial expressions, and movements to detect silence. If no speech is heard for a certain period of time (for example, 10 seconds), the server determines this as silence.

[0299] Step 4:

[0300] When the server detects silence, it also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. If a specific participant is losing concentration or showing negative emotions, it sends a notification encouraging them to speak. At this time, an appropriate message is selected depending on the situation.

[0301] Step 5:

[0302] The terminal displays the notification received from the server to the user. Specifically, a message such as "Please give your opinion next" is displayed on the screen of the user terminal.

[0303] Step 6:

[0304] Users can check the notifications displayed on their devices and express their opinions or questions. The content of their comments is captured by the device and sent to the server in real time.

[0305] Step 7:

[0306] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The server also reflects the analysis results of the emotion engine, and generates summary information that takes into account changes in emotions.

[0307] Step 8:

[0308] The server sends a summary of the conference to all participants' terminals, which receive the summary information and display it to the user, allowing the user to keep track of the conference progress.

[0309] Step 9:

[0310] The server records the content of comments and analysis data that is accumulated in real time, and stores the data necessary for creating minutes after the meeting.

[0311] Step 10:

[0312] After the meeting, the server automatically generates minutes based on the saved remarks and analysis data, including the remarks made by each speaker, key discussion points, and emotional state.

[0313] Step 11:

[0314] The server scores each participant based on the number and content of their comments, which reflects the quality, quantity, and emotional state of their comments.

[0315] Step 12:

[0316] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[0317] Step 13:

[0318] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement in the future.

[0319] These are the specific processing steps of this system, which combines an emotion engine. This system ensures smooth progress in meetings and highly productive discussions. Furthermore, by taking into account the emotional state of participants, the quality of meetings can be further improved.

[0320] Example 2

[0321] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0322] Conventional conferencing systems lack the means to effectively analyze participants' remarks, facial expressions, actions, and emotions in real time, resulting in issues such as the inability to smoothly progress through the meeting and the inability to respond to changes in participants' concentration and emotional state. Furthermore, there are insufficient means to prompt appropriate participants to speak when silence persists, resulting in a decrease in meeting efficiency. Furthermore, summarizing discussions during meetings and creating meeting minutes after the meeting are often done manually, which is time-consuming and labor-intensive.

[0323] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0324] In this invention, the server includes: a means for receiving video and audio streams during a conference; a means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; an emotion engine including means for determining participants' emotional states from facial expressions and tone of voice; a means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; a means for sending notifications encouraging participants who show a negative emotional state or a lack of concentration to speak; a means for summarizing the conference at regular intervals based on the content of comments made during the conference and the progress of the discussion; a means for sending the summarized conference summary to all participants' terminals; a means for automatically generating minutes after the conference based on all comments and analysis data; a means for scoring each participant based on the number of comments and the content of their comments; and a means for individually transmitting the scored results. This not only facilitates smooth progress in the conference, but also takes into account the participants' emotional states, enabling more effective discussions. Furthermore, the creation of summaries during the conference and the automatic generation of minutes after the conference ends significantly reduces time and effort.

[0325] 1. "Video Stream" means a data stream for transmitting and receiving video of participants in real time during a conference.

[0326] 2. "Audio Stream" means a data stream for transmitting and receiving the voices of participants in a conference in real time.

[0327] 3. "Analysis" refers to the process of decomposing and processing the content of received video and audio streams to extract various information.

[0328] 4. "Speech content" refers to the text information spoken by each participant during the meeting.

[0329] 5. “Facial expression” refers to information about a participant’s emotional state that can be read from their facial movements and expressions.

[0330] 6. "Movement" refers to information about participants' body movements and gestures.

[0331] 7. "Emotional state" refers to the psychological state of the participant as analyzed based on facial expressions and tone of voice.

[0332] 8. The "Emotion Engine" is software that analyzes and determines participants' emotional state from facial expressions and tone of voice.

[0333] 9. "Silence detection" is the process of automatically detecting periods of inactivity during a meeting.

[0334] 10. A "prompt to speak" is a message sent to prompt a specific participant to speak next when there is a period of silence.

[0335] 11. A "meeting summary" is a concise summary of the major topics and decisions discussed during the meeting.

[0336] 12. "Summarizing" refers to extracting important points based on what was said during a meeting and the progress of the discussion, and summarizing them in a concise form.

[0337] 13. "Minutes" means a document that contains a detailed record of what was discussed, what decisions were made, and what each speaker said at a meeting.

[0338] 14. "Scoring" is the process of evaluating each participant based on the number of comments and the quality of their comments.

[0339] 15. "Multimodal analysis" refers to the technique of simultaneously analyzing multiple data formats (e.g., audio and video).

[0340] 16. "Timer means" means a device or software that measures a certain period of time and monitors whether or not any speech is made during that period.

[0341] This invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. A specific embodiment of this system is described below.

[0342] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows for real-time analysis of participants' speech, facial expressions, actions, and emotions. Specifically, speech is converted into text using speech recognition technology. Facial and motion recognition algorithms are used to analyze facial expressions and actions. The emotion engine determines participants' emotional states from their facial expressions and tone of voice. Based on the results of this analysis, the server detects silence and prompts participants who show negative emotional states or low concentration to speak.

[0343] For example, if a 10-second silence occurs during a meeting, participants who are showing signs of low concentration or negative emotions can be notified by the emotion engine's analysis, with the message "Please speak next." The device that receives this notification will display it to the user, encouraging them to speak next.

[0344] The device displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the device captures the content and sends it to the server. It also captures the user's facial expressions and movements and sends them to the server. Furthermore, if the server sends a notification encouraging the user to speak or a summary of the conference, the device displays it to the user.

[0345] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0346] For example, the server summarizes the meeting every 15 minutes based on what is said during the meeting and the progress of the discussion, and sends this summary to the devices of all participants. If the meeting progresses on two topics, "proposing a new project" and "checking the budget," the server notifies the user of the summary. This allows users to always understand the progress of the meeting.

[0347] Next, after the meeting ends, the server automatically generates minutes based on the text data of comments and facial expression and emotion analysis data. For example, detailed minutes such as "Speaker A: Regarding the new project proposal" and "Speaker B: Opinions on the proposal" are created. Furthermore, each speaker is scored based on the number and content of their comments, taking into account the quality of their comments and their emotional state. Users can check the number of comments they made and the number of important opinions they expressed, and use this as an indicator to improve meeting performance.

[0348] Below are some examples of prompts to input to a generative AI model:

[0349] 1. When silence occurs during a meeting, notifications are sent to participants who are not concentrating or showing negative emotions to encourage them to speak up.

[0350] 2. As the meeting progresses, a summary of what was discussed is sent every 15 minutes to all participants' devices.

[0351] 3. After the meeting, minutes are automatically generated based on the content of the comments and the analysis data, and each speaker is scored, taking into account the results of the emotion engine.

[0352] In this way, the system facilitates smooth meeting progress and supports effective discussions that take into account the emotional state of participants.

[0353] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0354] Step 1:

[0355] Starting a meeting and capturing data

[0356] When a user presses the conference start button on the terminal, the terminal starts capturing the conference video and audio streams. These streams are sent to the server as they are. The input is video and audio data from the camera and microphone connected to the terminal, and the output is sent to the server as stream data in real time.

[0357] Specifically, when a user presses a button to join a conference, the camera captures video and the microphone captures audio, and then these are sent to the server.

[0358] Step 2:

[0359] Video and audio stream analysis

[0360] The server analyzes the received video and audio streams using a generative AI model, converting speech into text using speech recognition technology (input: audio stream, output: text data), and analyzing facial expressions and movements using facial and movement recognition algorithms (input: video stream, output: facial and movement data).

[0361] Specifically, when a user says, "What do you think about this proposal?", the voice data is converted into text and saved as text data saying, "What do you think about this proposal?" At the same time, the user's facial expressions and movements are extracted from the video stream and converted into data.

[0362] Step 3:

[0363] Emotional state analysis using emotion engine

[0364] The server uses an emotion engine to determine the participant's emotional state from the analyzed facial expressions and tone of voice. The emotion engine categorizes the emotion from the analysis results (input: facial expression and tone of voice data, output: emotional state data) and uses that data for the next step.

[0365] Specifically, the emotion engine analyzes the facial expressions and tone of voice of the user when they speak, and converts emotions such as "joy," "anger," and "surprise" into data.

[0366] Step 4:

[0367] Silence detection and prompting

[0368] The server monitors the audio data during the conference and detects silence for a certain period of time (for example, 10 seconds) (input: audio data, output: silence detection status). When silence is detected, participants with negative emotional states or participants with declining concentration are identified based on the analysis results of the emotion engine, and a notification is sent to those participants encouraging them to speak (input: emotional state data, output: notification encouraging them to speak).

[0369] Specifically, the server detects 10 seconds of silence and sends a notification to participants whose emotion engine indicates a negative emotional state, saying, "Please give your opinion next."

[0370] Step 5:

[0371] Meeting summary and distribution

[0372] The server summarizes the discussion every 15 minutes. Using a generative AI model, it extracts key topics and decisions from each comment and creates a summary (input: comment data, output: summary data). This is then sent to all participants' devices.

[0373] Specifically, the server summarizes the contents of the 15-minute meeting into the topics "proposing a new project" and "confirming the budget," and distributes the summary information to the terminals of all participants.

[0374] Step 6:

[0375] Automatic generation and scoring of meeting minutes after the meeting

[0376] After the meeting ends, the server automatically generates minutes based on the accumulated remarks and analysis data. Using a generative AI model, the server records the speaker's name, remarks, timing of remarks, and emotional state in detail (input: remarks and analysis data, output: minutes data). Furthermore, it scores each speaker based on the number and content of their remarks (input: remarks data, output: scoring data). This is then sent to each participant's device.

[0377] Specifically, the server creates detailed minutes such as "Speaker A: Regarding the proposal for a new project" and "Speaker B: Comments on the proposal," and then notifies each participant of the scoring results, such as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments."

[0378] (Application example 2)

[0379] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] The issues of declining work efficiency and ensuring worker safety in factories are extremely important. Conventional systems have difficulty monitoring and analyzing the progress of work, the concentration level of workers, and their emotional state in real time, and issuing appropriate instructions or warnings as needed. Another issue is that while efforts to improve work efficiency have been made, there has been a lack of efforts to promote breaks that take into account workers' fatigue and emotional state.

[0381] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0382] In this invention, the server includes: means for receiving video and audio streams during a conference; means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; means for sending notifications to encourage participants to speak; means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion; means for sending the summarized conference summaries to all participants' terminals; means for automatically generating minutes after the conference based on all comments and analysis data; means for scoring each participant based on the number of comments and the content of their comments; means for individually sending the scored results based on the number of comments and the content of their comments; means for monitoring work in real time and issuing a warning when a specific movement is detected; means for analyzing the concentration and emotional state of workers and encouraging them to take a break as necessary; and means for instructing the next work step when efficiency drops, thereby improving work efficiency in factories and ensuring worker safety.

[0383] A "video stream" refers to video data that is continuously sent from a device such as a camera.

[0384] An "audio stream" refers to a continuous stream of audio data sent from a microphone or other audio input device.

[0385] The term "analysis means" refers to a method or device for analyzing the content of speech, facial expressions, and actions of participants based on the video stream and audio stream.

[0386] "Means for detecting silence" refers to a method or device for detecting a period of no speech.

[0387] "Notification means" refers to a method or device for notifying participants of analysis results, warnings, instructions, etc.

[0388] "Means for summarizing the meeting" refers to a method or device for periodically summarizing what has been said and the progress of discussions during a meeting.

[0389] "Means for automatically generating minutes" refers to a method or device for creating minutes after a meeting based on the content of remarks and analysis data.

[0390] "Scoring means" refers to a method or device for evaluating each participant based on the number of comments and the content of their comments.

[0391] "Real-time monitoring means" refers to a method or device for monitoring and analyzing the real-time progress of work.

[0392] "Alert means" refers to a method or device for issuing an alert when a particular action or condition is detected.

[0393] "Means to encourage breaks" refers to methods or devices for analyzing a worker's concentration level and emotional state and instructing them to take a break as necessary.

[0394] "Means for indicating work steps" refers to a method or device for indicating the next work procedure when work efficiency drops.

[0395] The present invention relates to a work supervision system that aims to improve work efficiency in a factory and ensure the safety of workers. A specific embodiment of this system will be described below.

[0396] System configuration

[0397] Hardware configuration:

[0398] Smart glasses: Devices worn by workers that capture visual and audio information.

[0399] Server: A high-performance computer that analyzes collected data and generates instructions and warnings.

[0400] Camera: A device installed in each work area in a factory that provides a video stream.

[0401] Microphone: A device used to capture the voice of workers.

[0402] Software configuration:

[0403] Generative AI model (e.g., GPT-3): Converts input voice data into text and analyzes what is being said.

[0404] Emotion analysis engine (e.g., Emotion API): Analyzes facial expressions and movements from input video data to determine emotional state.

[0405] Streaming processing software (e.g., WebRTC): Sends and receives video and audio streams in real time.

[0406] Program processing

[0407] The server receives video and audio streams from the smart glasses and camera. The received data is analyzed in real time to analyze the worker's speech, facial expressions, and movements. This analysis is performed using a generative AI model and an emotion analysis engine. Based on the analysis results, the following processes are performed:

[0408] 1. Real-time monitoring: There is a way to monitor the progress of work and issue alerts when certain conditions are met (e.g., not wearing proper protective equipment).

[0409] 2. Promoting breaks: Analyze workers' concentration levels and emotional states and encourage them to take breaks as needed.

[0410] 3. Efficiency: There is a way to indicate the next work step if work efficiency declines.

[0411] Specific examples

[0412] Example 1: When worker A's concentration level is low

[0413] Based on the results of the server's analysis, the smart glasses will notify the user, "Your concentration level is declining. Please take a break."

[0414] Example 2: When worker B is not wearing protective equipment

[0415] The smart glasses notify workers to wear protective equipment, encouraging them to work safely.

[0416] Prompt Sentence Examples

[0417] Example prompt for generative AI model:

[0418] "Design an AI system to manage the work of the next shift. To ensure worker safety and efficiency, the system should include the following elements: facial expression analysis, voice recognition, emotion analysis, and real-time warning instructions. The following are specific prompts for analysis. The proposed configuration includes: when a worker is not wearing protective gear, when work efficiency drops sharply, how to issue appropriate instructions, how to communicate data between the server and the smart glasses, and what to display on the smart glasses."

[0419] The above is a specific embodiment of the present invention. This system realizes improved efficiency and safety in factory work.

[0420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0421] Step 1:

[0422] The smart glasses and camera capture video and audio streams and send them to a server.

[0423] Input: Video and audio streams from smart glasses and camera.

[0424] Output: Sending video and audio streams to the server.

[0425] Step 2:

[0426] The server analyzes the received video and audio streams using a generative AI model and emotion analysis engine.

[0427] Input: Video and audio streams.

[0428] Output: Analysis data on each participant's speech, facial expressions, and movements.

[0429] Specific operations: Converting voice data into text and analyzing spoken content using a generative AI model, and analyzing facial expressions and movements using an emotion analysis engine.

[0430] Step 3:

[0431] The server monitors the progress of work based on the analysis results and detects specific actions or conditions.

[0432] Input: Analysis data.

[0433] Output: The result of detecting a specific behavior or state.

[0434] Specific actions: Checking that workers are wearing the correct protective equipment, detecting facial expressions such as fatigue or impatience.

[0435] Step 4:

[0436] The server sends an alert notification to the smart glasses when certain conditions are met.

[0437] Input: The detection of a specific action or state.

[0438] Output: Alert notification to smart glasses.

[0439] Specific operation: If a worker is not wearing protective equipment, the system notifies the worker to "put on protective equipment."

[0440] Step 5:

[0441] The server analyzes the worker's concentration level and emotional state and encourages them to take a break if necessary.

[0442] Input: Analysis data.

[0443] Output: Break promotion notification to smart glasses.

[0444] Specific behavior: If a worker's concentration level is decreasing, the system will notify them, "Your concentration level is decreasing. Please take a break."

[0445] Step 6:

[0446] The server instructs the next work step when the work efficiency drops.

[0447] Input: Data about work progress and efficiency.

[0448] Output: Next work step instructions to the smart glasses.

[0449] Specific behavior: If work is stalled, notify the user to "proceed to the next work step."

[0450] Through the above steps, efficiency and safety of work in the factory are improved.

[0451] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0452] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0453] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0454] [Second embodiment]

[0455] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0456] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0457] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0458] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0459] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0460] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0461] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0462] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0463] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0464] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0465] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0466] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0467] The system of the present invention is designed to analyze the video and audio streams of a conference in real time and facilitate the progress of the conference. A specific embodiment of this system will be described below.

[0468] The server receives the video and audio streams of the conference. It then analyzes the received video and audio streams to analyze the participants' speech, facial expressions, and movements in real time. Based on the analysis results, the server detects silence and identifies participants who should be encouraged to speak. It also sends notifications to encourage them to speak.

[0469] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The summarized meeting summary is sent to the devices of all participants. After the meeting ends, minutes are automatically generated based on all comments and analysis data. Each participant is scored based on the number of comments and the content of their comments, and the results are sent individually.

[0470] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[0471] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0472] The operation of the system will be explained below using specific examples.

[0473] Example 1: Silence detection and prompting

[0474] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be prompted to speak next. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who then gives their opinion based on it.

[0475] Example 2: Meeting summary and sharing

[0476] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[0477] Example 3: Minutes and scoring

[0478] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[0479] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[0480] The processing flow will be explained below.

[0481] Step 1:

[0482] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[0483] Step 2:

[0484] The server analyzes the received video and audio streams. Specifically, it uses a generative AI model to analyze participants' speech, facial expressions, and movements in real time. Speech is converted into text, and facial expressions and movements are digitized as analysis results.

[0485] Step 3:

[0486] The server detects silence during the conference based on the analysis results. If no speech is made for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings.

[0487] Step 4:

[0488] When the server detects silence, it prompts the most appropriate participant to speak. It identifies participants who are not concentrating or appear sleepy from the analysis results and sends them a notification to prompt them to speak.

[0489] Step 5:

[0490] The device displays the notification received from the server to the user. Specifically, it displays a message such as "Please state your opinion next" on the screen to encourage the user to speak.

[0491] Step 6:

[0492] The user checks the notification displayed on the device and follows the instructions to express their opinions or ask questions. The comments are then sent back to the server via the device.

[0493] Step 7:

[0494] The server summarizes the meeting at regular intervals (for example, 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. This summary is generated in text format.

[0495] Step 8:

[0496] The server sends a summary of the meeting to all participants' terminals, which receive the summary message and display it to the user.

[0497] Step 9:

[0498] As the meeting progresses, the server continuously records what is said and the analytical data, so that all comments and discussions are stored in a log.

[0499] Step 10:

[0500] After the meeting, the server automatically generates minutes based on the accumulated remarks and analysis data. Specifically, it organizes the remarks made by each speaker and important points of discussion.

[0501] Step 11:

[0502] The server scores each participant based on the number and content of their comments. For example, the evaluation score is calculated based on multiple indicators such as the number of comments, the quality of the comments, and the degree of contribution to the discussion.

[0503] Step 12:

[0504] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[0505] Step 13:

[0506] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement.

[0507] The above are the specific processing steps of the present system based on the claims. This system allows meetings to proceed smoothly and enables highly productive discussions.

[0508] Example 1

[0509] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0510] Conventional conference systems lack sufficient support functions to facilitate smooth meeting progress, making it difficult for all participants to participate effectively in discussions. Furthermore, recording comments, generating minutes, and analyzing the content of comments must be done manually, which requires a great deal of time and effort. Another issue is that there are few ways to grasp participants' own contributions to the discussion, making it difficult to obtain specific feedback to improve the quality and efficiency of the meeting.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0512] In this invention, the server includes means for receiving video data and audio data during a conference, means for analyzing the received video data and audio data and analyzing the participants' comments, facial expressions, and movements in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion, means for transmitting the summarized conference summaries to all participants' terminals, means for automatically generating minutes after the conference based on all comments and the analysis data, means for scoring each participant based on the number of comments and the content of their comments, means for individually transmitting the scoring results based on the number of comments and the content of their comments, means for using computer vision technology to analyze the video data, means for using voice recognition technology to analyze the audio data, means for monitoring the audio level to detect silence for a certain period of time, means for notifying all participants of the generated conference summary, and means for distributing the minutes and scoring results to all participants.This allows the conference to proceed smoothly and allows all participants to participate efficiently in the discussion. In addition, automated recording and analysis of meetings reduces effort and time, and allows each participant to understand their own contribution and receive specific feedback that can help improve their meeting skills.

[0513] "Video data during a meeting" refers to visual information acquired through a camera while the meeting is in progress, and includes visual content such as the movements and facial expressions of participants.

[0514] "Audio data" refers to acoustic information acquired through a microphone during a conference, including speeches made by participants and other audio information.

[0515] "Means for receiving" refers to devices or software that provide the function of acquiring video data and audio data and inputting them into the server.

[0516] "Means for analyzing" refers to an algorithm or computer program that processes the received data and extracts specific information.

[0517] "Real-time analysis" refers to the act of processing acquired data immediately to obtain results, providing information immediately without delay.

[0518] "Silence detection means" means a device or software that provides the functionality to identify and detect periods of silence within audio data.

[0519] "Means for identifying participants who should be encouraged to speak" refers to algorithms or computer programs that use the analyzed data to identify the next person to speak.

[0520] "Means for sending notifications" refers to machines and programs for sending messages or alerts to specific participants.

[0521] A "meeting summary tool" refers to an algorithm or computer program that concisely summarizes what was discussed during a meeting.

[0522] "Means for sending a conference summary to all participants' terminals" refers to machines and programs that distribute messages to convey summarized information to all participants.

[0523] "Means for automatically generating minutes" refers to algorithms or computer programs that automatically create detailed records of meetings based on what was said during the meeting and analytical data.

[0524] "Means for scoring" refers to an algorithm or computer program that evaluates the number and content of each participant's comments and calculates a score based on that.

[0525] "Means for individual transmission" refers to a machine and program that provides the functionality for transmitting scoring results to each participant's terminal in a specific format.

[0526] "Computer vision technology" refers to a collection of techniques and algorithms used to extract information from digital images and video.

[0527] "Speech recognition technology" refers to a collection of techniques and algorithms for analyzing voice data and converting it into text.

[0528] "Means for monitoring audio levels" refers to an algorithm or computer program that continuously monitors the volume of audio data and detects fluctuations in the volume.

[0529] The system of the present invention is designed to analyze the video and audio streams of a conference in real time to facilitate the progress of the conference. A specific embodiment of this system will be described below, with each role played by a server, a terminal, and a user.

[0530] Server Roles

[0531] The server processes data and provides functions to support the conference through the following procedures:

[0532] 1. Receiving data:

[0533] The server receives video and audio data during the conference using a communication protocol such as WebRTC, which transmits real-time video and audio streams from each participant to the server.

[0534] 2. Video and audio data analysis:

[0535] The received video data is analyzed using OpenCV and other computer vision technologies, specifically using facial recognition and expression analysis algorithms to understand the participants' facial expressions and movements.

[0536] The voice data is converted into text using voice recognition technology (for example, Google Cloud Speech-to-Text API), which makes the spoken content available as text data.

[0537] 3. Silence detection and prompting:

[0538] The server continuously monitors the audio level and detects silence for a certain period (for example, 10 seconds) using a specific algorithm that monitors the audio level.

[0539] When silence is detected, an algorithm determines who should be the next person to speak and sends them a notification, such as "You're next."

[0540] 4. Summary of the meeting:

[0541] Based on what was said during the meeting and the progress of the discussion, a generative AI model is used to summarize the main topics and decisions of the meeting at regular intervals (for example, every 15 minutes).

[0542] A summary of the meeting is sent to all participants' devices, making it easier for them to understand the flow of the meeting.

[0543] 5. Generate transcripts:

[0544] After the meeting, minutes are automatically generated based on all statements and analysis data. The generated minutes include a detailed record of what each speaker said.

[0545] 6. Scoring:

[0546] Each participant is scored based on the number of comments and the content of their comments. This scoring is based on the evaluation criteria of the frequency, importance, and contribution of comments.

[0547] The scored results are sent individually to each participant's device.

[0548] Device Role

[0549] The terminal provides the user with visual and auditory information about the data sent from the server, and transmits user operations to the server.

[0550] 1. Displaying data:

[0551] The terminal displays the video and audio streams sent from the server, allowing users to view the contents of the conference in real time.

[0552] 2. Capturing what users say and do:

[0553] When a user speaks, the device captures the user's voice and sends it to the server. It also captures the user's facial expressions and movements and sends this data to the server for analysis.

[0554] 3. Notification and Summary Display:

[0555] When a notification encouraging speaking or a summary of the conference is sent from the server, the terminal displays this to the user as a pop-up notification or message.

[0556] User Roles

[0557] Users can participate in the conference and make comments and hold discussions as usual. In addition, they can check notifications and summaries from the server and use them to express their opinions and advance the discussion.

[0558] 1. Statement:

[0559] The user speaks through a microphone, and the content of the speech is transmitted to the server via the terminal.

[0560] Facial expressions and movements are also captured by the camera and sent to the server.

[0561] 2. Responding to Notices:

[0562] When a user receives a notification from the server prompting the user to speak, the user responds by speaking.

[0563] 3. Review of minutes and scoring results:

[0564] After the meeting, users can check the minutes and scoring results displayed on their device to help improve their own meeting skills.

[0565] Specific examples

[0566] Example 1: Silence detection and prompting

[0567] During a meeting, if a participant finishes speaking and there is a 10-second silence, the server detects the silence and identifies the next participant to be prompted to speak. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who can then express their opinion.

[0568] Example 2: Meeting Summary

[0569] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[0570] Example 3: Minutes and scoring

[0571] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[0572] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[0573] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] Receiving data

[0576] The server receives video and audio data during the conference using the WebRTC protocol.

[0577] Input: Real-time video and audio streams sent from each participant's device.

[0578] Data processing: Video and audio data are received in the form of packets.

[0579] Output: Received video and audio data.

[0580] How it works: The server uses the WebRTC protocol to receive video and audio streams from each participant in real time, store them in a database, and queue them for processing.

[0581] Step 2:

[0582] Video data analysis

[0583] The server analyzes the received video data using the OpenCV library.

[0584] Input: Video data received via the WebRTC protocol.

[0585] Data processing: Applying facial recognition and expression analysis algorithms to extract specific facial features and changes in facial expressions.

[0586] Output: Participant's facial expression and movement data.

[0587] Specific operation: The server uses OpenCV to detect facial features for each video frame and analyzes changes in facial expression and eye movements, thereby classifying facial expressions into categories such as "smiling," "serious," and "sleepy."

[0588] Step 3:

[0589] Analysis of audio data

[0590] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0591] Input: Audio data received via the WebRTC protocol.

[0592] Data processing: Using voice recognition technology, voice data is converted into text data.

[0593] Output: Text data of the speech.

[0594] What it does: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts what is said into text in real time, which is then stored in a database for further analysis and recording.

[0595] Step 4:

[0596] Silence Detection

[0597] The server monitors the analyzed audio data and detects periods of silence.

[0598] Input: Speech and speech level data converted to text.

[0599] Data processing: Based on the voice level data, monitor whether there is any continuous speech for a specific period (for example, 10 seconds).

[0600] Output: Silence detection results.

[0601] What happens: The server checks that the audio level is below a certain threshold and then runs an algorithm that detects silence if no speech is heard for 10 consecutive seconds.

[0602] Step 5:

[0603] Sending a call to speak

[0604] If a period of silence continues, the server identifies the participant who should next be prompted to speak and sends a notification.

[0605] Input: Silence detection results and analyzed facial expression data.

[0606] Data processing: Based on the analysis results obtained from video and audio data, participants who should be encouraged to speak are identified.

[0607] Output: Informational message.

[0608] Specific operation: The server sends a notification to the identified participant's device saying, "Please give your opinion next." This notification is displayed as a pop-up message on the device.

[0609] Step 6:

[0610] Conference Summary

[0611] The server periodically summarizes the contents of the conference.

[0612] Input: Text data converted from audio data and discussion progress.

[0613] Data processing: Using NLP techniques, we analyze what is being said and extract key topics and decisions.

[0614] Output: Meeting summary text.

[0615] Specific operation: The server uses a generative AI model to summarize the contents of the meeting every 15 minutes and sends the summary text to the devices of all participants.

[0616] Step 7:

[0617] Generate meeting minutes

[0618] After the meeting ends, the server automatically generates minutes based on all statements and analysis data.

[0619] Input: Text data and analysis data of all statements.

[0620] Data processing: Integrate the statements and analytical data and organize them into detailed records.

[0621] Output: Auto-generated meeting transcript.

[0622] Specific operation: The server automatically generates minutes based on all data collected during the meeting and sends them to each participant's device. The minutes include the remarks and decisions made by each participant.

[0623] Step 8:

[0624] Sending scoring results

[0625] The server scores each participant based on the number of comments and the content of their comments, and sends the results individually.

[0626] Input: All speech content and number of speeches data.

[0627] Data processing: A score is calculated for each speaker based on the number of comments, importance, and contribution of each speaker.

[0628] Output: Scoring results.

[0629] How it works: The server evaluates and scores each participant's performance, taking into account the frequency of their comments and the importance of their content. The results are sent to each participant's device as individual notifications.

[0630] These are the specific processing steps of the system. This allows meetings to proceed smoothly and enables all participants to participate in the discussion efficiently. In addition, the automatic generation of meeting minutes and feedback of scoring results can contribute to improving the quality of meetings and the skills of participants.

[0631] (Application example 1)

[0632] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Conventional conference systems have difficulty efficiently and effectively managing tasks such as encouraging participants to speak and creating meeting minutes. In particular, current conference management systems are insufficient in production sites such as factories, where rapid decision-making and communication are required. Furthermore, there is no system for analyzing participants' facial expressions and movements in real time, making it difficult to properly grasp the status of participants during the meeting, which can slow down the progress of the meeting. Therefore, an effective system to solve these issues is needed.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0635] In this invention, the server includes means for receiving video and audio streams during a conference, means for analyzing the received video and audio streams and analyzing the speech content, facial expressions, and movements of participants in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for analyzing the video and audio streams and encouraging specific participants to speak to support in-factory conferences, and means for applying a facial expression analysis model using the video and audio streams, identifying facial expressions based on the analysis results, and determining participants who should be encouraged to speak, thereby enabling quick decision-making and efficient communication within the factory.

[0636] A "video stream" is a series of video data captured during a conference via a video input device such as a camera.

[0637] An "audio stream" is a series of audio data captured during a conference via an audio input device such as a microphone.

[0638] "Real-time analysis" refers to the process of analyzing captured video and audio streams in real time and providing results without delay.

[0639] "Silence detection" refers to determining whether a silence state has occurred for a certain period of time or longer based on the analyzed audio stream.

[0640] "Speaker prompt notification" is a function that sends a message to specific participants to prompt them to speak when silence is detected.

[0641] The "facial expression analysis model" is a machine learning model for analyzing participants' facial expressions obtained from video streams and estimating their emotional states.

[0642] The "analysis results" are information on speech content, facial expressions, and movements obtained from the video stream and audio stream data.

[0643] "Scoring" is the process of evaluating each participant based on the number and content of their comments.

[0644] Minutes are documents that record what was said and what was decided at a meeting.

[0645] "Factory meetings" are meetings held in factories for the purpose of planning and solving problems related to production.

[0646] "Notification means" is a function for sending messages and instructions from the system to participants.

[0647] "Multimodal analysis" is the process of simultaneously analyzing multiple different data sources (such as video and audio streams) and utilizing the results in a comprehensive manner.

[0648] The present invention provides a system for smoothly conducting meetings in a factory and promoting efficient decision-making. Specific embodiments for implementing this system will be described below.

[0649] System Overview

[0650] This system consists of three roles: a server, a terminal, and a user. The server plays a central role in receiving and analyzing the video and audio streams of the conference. The terminal is a device through which participants interact with the system, displaying the video and audio streams and receiving notifications. The user participates in the conference, expressing their opinions and participating in discussions.

[0651] Hardware and Software Configuration

[0652] Hardware: Camera, microphone, display, smartphone.

[0653] Software: Python, OpenCV (video stream processing), TensorFlow (facial expression analysis model), Numpy (data processing).

[0654] Data processing and calculation

[0655] Server Processing

[0656] 1. Receiving video and audio streams

[0657] The server receives the video and audio streams from the camera and microphone in real time.

[0658] 2. Real-time analysis

[0659] We use OpenCV to detect participants' faces from the video stream, TensorFlow to analyze detected facial expressions and estimate participants' emotional states, and Numpy to analyze the volume levels of the audio stream and detect silence.

[0660] 3. Detect silence and send notifications

[0661] If silence continues for a certain period of time, the system will identify the participant who should be prompted to speak next based on the analysis results and send a notification to that participant, which will be displayed on their device.

[0662] 4. Meeting summary and minutes generation

[0663] Based on the main comments made during the meeting and the progress of the discussion, a summary of the meeting is generated at regular intervals (for example, every 15 minutes) and sent to the devices of all participants. After the meeting, minutes are automatically generated based on all comments and analysis data, and the number and content of comments made by each participant are scored.

[0664] Terminal handling

[0665] 1. Displaying video and audio streams

[0666] The terminal displays the video and audio streams sent from the server, allowing participants to visually understand the contents of the conference.

[0667] 2. Receiving and Displaying Notifications

[0668] It receives notifications from the server and displays them to users, who can then express their opinions and advance the discussion.

[0669] Specific examples

[0670] If there is silence during a meeting, the system will:

[0671] 1. At the beginning of the meeting, the statement is made, "We will begin discussing the design of the new product."

[0672] 2. If there is a period of silence, the server detects this and identifies the next participant to speak.

[0673] 3. A notification will appear on the device of the person you are speaking to saying, "Yamada-san, please comment on the next steps."

[0674] 4. This allows Yamada to express his opinion and the meeting proceeds smoothly.

[0675] An example prompt is:

[0676] Meeting start: "We're going to start discussing the design of our new product."

[0677] There is silence for a while.

[0678] Prompt to speak: "Yamada-san, please comment on next steps."

[0679] In this way, this system enables smooth progress in factory meetings and efficient decision-making.

[0680] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0681] Step 1:

[0682] Receiving video and audio streams

[0683] The server receives video and audio streams in real time through cameras and microphones. The input is the video and audio data of the meeting, and the output is the raw data stored and prepared for analysis. In this step, the server continuously acquires data by capturing the streams from each device.

[0684] Step 2:

[0685] Real-time analysis of video streams

[0686] The server uses OpenCV to detect participants' faces from the video stream. The input is the received video stream data, and the output is the location information of the detected faces. This face location information becomes the input for subsequent facial expression analysis. Specifically, the server recognizes the contours of the face from the video frame and identifies its location.

[0687] Step 3:

[0688] Application of facial expression analysis model

[0689] The server performs facial expression analysis on the detected faces using TensorFlow. The input is image data of salient faces and a pre-trained facial expression analysis model, and the output is an estimated emotional state (e.g., joy, anger, sadness, happiness, etc.). This is the process of applying the facial expression analysis model to classify facial expressions.

[0690] Step 4:

[0691] Parsing an Audio Stream

[0692] The server uses Numpy to analyze the volume level of the audio stream and detect silence. The input is the audio stream data acquired in real time, and the output is the volume level (silence or not) judgment result. Specifically, it calculates the amplitude of the audio data and determines silence if it is below a preset threshold.

[0693] Step 5:

[0694] Detect silence and send prompts

[0695] If silence continues for a certain period of time (e.g., 10 seconds), the server identifies the participant who should be prompted to speak next based on the results of facial expression analysis. The input is the silence detection result and the results of facial expression analysis, and the output is the identified participant ID and the corresponding prompt to speak. The server then sends a "please speak next" notification to this identified participant.

[0696] Step 6:

[0697] Viewing video and audio streams

[0698] The terminal receives the video and audio streams sent from the server and displays them to the user. The input is the video and audio stream data from the server, and the output is the display of that data on the terminal. This allows the user to grasp the status of the conference in real time.

[0699] Step 7:

[0700] Receiving and Viewing Notifications

[0701] The terminal receives a prompt to speak from the server and displays it to the user. The input is the prompt to speak notification data from the server, and the output is the display of the notification to the user. Upon receiving this notification, the user can express their opinion or advance the discussion.

[0702] Step 8:

[0703] Generate meeting summaries and minutes

[0704] The server generates a summary of the meeting at regular intervals (e.g., every 15 minutes) based on the main comments made during the meeting and the progress of the discussion, and sends it to the devices of all participants. After the meeting ends, it automatically generates minutes based on all comments and analysis data, scoring the number and content of comments made by each participant. The input is all comment data and analysis results from the meeting, and the output is summary information and minutes. Specifically, it uses natural language processing technology to create summaries, extract necessary information, and document it.

[0705] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0706] The present invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. Specific embodiments of this system are described below.

[0707] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows participants' speech, facial expressions, actions, and emotions to be analyzed in real time. Speech is converted into text, and facial expressions and actions are converted into data as analysis results. The emotion engine also recognizes participants' emotional states from their facial expressions and tone of voice.

[0708] Based on the analysis results, the server detects silence during the meeting. If no one speaks for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings. When silence is detected, the server also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. For example, it sends a notification encouraging participants who are determined to be not concentrating or whose emotional state is changing negatively to speak.

[0709] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[0710] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0711] The operation of the system will be explained below using specific examples.

[0712] Example 1: Silence detection and prompting

[0713] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be encouraged to speak next. If the emotion engine's analysis determines that a particular participant is losing concentration or showing negative emotions, it sends that participant a notification saying, "Please speak next." The device displays this notification to the user, who then expresses their opinion based on it.

[0714] Example 2: Meeting summary and sharing

[0715] As the meeting progresses, the server summarizes the content of the discussion every 15 minutes. For example, it briefly summarizes the main topics and decisions discussed in the first 15 minutes and sends this content to the devices of all participants. The analysis results of the emotion engine are also taken into account, and summary information corresponding to increases in negative emotions and decreases in positive emotions is provided. The device displays this summary information to the user, allowing them to always understand the progress of the meeting.

[0716] Example 3: Minutes and scoring

[0717] After the meeting ends, the server automatically generates minutes based on the accumulated content of comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding a new project proposal" and "Speaker B: Opinions on the proposal." The analysis results of the emotion engine are also reflected, and the emotional state of each speaker is included in the minutes. In addition, each speaker is scored based on the number of comments and content of their comments. For example, they might be evaluated as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments." The scoring also takes into account the quality of the comments and the emotional state. These results are sent individually to each participant's device, allowing users to check their own performance.

[0718] The above is a specific embodiment of the present invention. This system not only allows meetings to proceed smoothly, but also allows for more effective discussions by taking into account the emotional states of participants. The introduction of an emotion engine will improve the quality of meetings and also help improve individual meeting skills.

[0719] The processing flow will be explained below.

[0720] Step 1:

[0721] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[0722] Step 2:

[0723] The server uses generative AI models and emotion engines to analyze the incoming video and audio streams in real time, converting participants' speech into text and digitizing their facial expressions, movements, and emotional states.

[0724] Step 3:

[0725] Based on the analysis results, the server monitors speech content, facial expressions, and movements to detect silence. If no speech is heard for a certain period of time (for example, 10 seconds), the server determines this as silence.

[0726] Step 4:

[0727] When the server detects silence, it also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. If a specific participant is losing concentration or showing negative emotions, it sends a notification encouraging them to speak. At this time, an appropriate message is selected depending on the situation.

[0728] Step 5:

[0729] The terminal displays the notification received from the server to the user. Specifically, a message such as "Please give your opinion next" is displayed on the screen of the user terminal.

[0730] Step 6:

[0731] Users can check the notifications displayed on their devices and express their opinions or questions. The content of their comments is captured by the device and sent to the server in real time.

[0732] Step 7:

[0733] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The server also reflects the analysis results of the emotion engine, and generates summary information that takes into account changes in emotions.

[0734] Step 8:

[0735] The server sends a summary of the conference to all participants' terminals, which receive the summary information and display it to the user, allowing the user to keep track of the conference progress.

[0736] Step 9:

[0737] The server records the content of comments and analysis data that is accumulated in real time, and stores the data necessary for creating minutes after the meeting.

[0738] Step 10:

[0739] After the meeting, the server automatically generates minutes based on the saved remarks and analysis data, including the remarks made by each speaker, key discussion points, and emotional state.

[0740] Step 11:

[0741] The server scores each participant based on the number and content of their comments, which reflects the quality, quantity, and emotional state of their comments.

[0742] Step 12:

[0743] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[0744] Step 13:

[0745] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement in the future.

[0746] These are the specific processing steps of this system, which combines an emotion engine. This system ensures smooth progress in meetings and highly productive discussions. Furthermore, by taking into account the emotional state of participants, the quality of meetings can be further improved.

[0747] Example 2

[0748] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0749] Conventional conferencing systems lack the means to effectively analyze participants' remarks, facial expressions, actions, and emotions in real time, resulting in issues such as the inability to smoothly progress through the meeting and the inability to respond to changes in participants' concentration and emotional state. Furthermore, there are insufficient means to prompt appropriate participants to speak when silence persists, resulting in a decrease in meeting efficiency. Furthermore, summarizing discussions during meetings and creating meeting minutes after the meeting are often done manually, which is time-consuming and labor-intensive.

[0750] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0751] In this invention, the server includes: a means for receiving video and audio streams during a conference; a means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; an emotion engine including means for determining participants' emotional states from facial expressions and tone of voice; a means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; a means for sending notifications encouraging participants who show a negative emotional state or a lack of concentration to speak; a means for summarizing the conference at regular intervals based on the content of comments made during the conference and the progress of the discussion; a means for sending the summarized conference summary to all participants' terminals; a means for automatically generating minutes after the conference based on all comments and analysis data; a means for scoring each participant based on the number of comments and the content of their comments; and a means for individually transmitting the scored results. This not only facilitates smooth progress in the conference, but also takes into account the participants' emotional states, enabling more effective discussions. Furthermore, the creation of summaries during the conference and the automatic generation of minutes after the conference ends significantly reduces time and effort.

[0752] 1. "Video Stream" means a data stream for transmitting and receiving video of participants in real time during a conference.

[0753] 2. "Audio Stream" means a data stream for transmitting and receiving the voices of participants in a conference in real time.

[0754] 3. "Analysis" refers to the process of decomposing and processing the content of received video and audio streams to extract various information.

[0755] 4. "Speech content" refers to the text information spoken by each participant during the meeting.

[0756] 5. “Facial expression” refers to information about a participant’s emotional state that can be read from their facial movements and expressions.

[0757] 6. "Movement" refers to information about participants' body movements and gestures.

[0758] 7. "Emotional state" refers to the psychological state of the participant as analyzed based on facial expressions and tone of voice.

[0759] 8. The "Emotion Engine" is software that analyzes and determines participants' emotional state from facial expressions and tone of voice.

[0760] 9. "Silence detection" is the process of automatically detecting periods of inactivity during a meeting.

[0761] 10. A "prompt to speak" is a message sent to prompt a specific participant to speak next when there is a period of silence.

[0762] 11. A "meeting summary" is a concise summary of the major topics and decisions discussed during the meeting.

[0763] 12. "Summarizing" refers to extracting important points based on what was said during a meeting and the progress of the discussion, and summarizing them in a concise form.

[0764] 13. "Minutes" means a document that contains a detailed record of what was discussed, what decisions were made, and what each speaker said at a meeting.

[0765] 14. "Scoring" is the process of evaluating each participant based on the number of comments and the quality of their comments.

[0766] 15. "Multimodal analysis" refers to the technique of simultaneously analyzing multiple data formats (e.g., audio and video).

[0767] 16. "Timer means" means a device or software that measures a certain period of time and monitors whether or not any speech is made during that period.

[0768] This invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. A specific embodiment of this system is described below.

[0769] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows for real-time analysis of participants' speech, facial expressions, actions, and emotions. Specifically, speech is converted into text using speech recognition technology. Facial and motion recognition algorithms are used to analyze facial expressions and actions. The emotion engine determines participants' emotional states from their facial expressions and tone of voice. Based on the results of this analysis, the server detects silence and prompts participants who show negative emotional states or low concentration to speak.

[0770] For example, if a 10-second silence occurs during a meeting, participants who are showing signs of low concentration or negative emotions can be notified by the emotion engine's analysis, with the message "Please speak next." The device that receives this notification will display it to the user, encouraging them to speak next.

[0771] The device displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the device captures the content and sends it to the server. It also captures the user's facial expressions and movements and sends them to the server. Furthermore, if the server sends a notification encouraging the user to speak or a summary of the conference, the device displays it to the user.

[0772] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0773] For example, the server summarizes the meeting every 15 minutes based on what is said during the meeting and the progress of the discussion, and sends this summary to the devices of all participants. If the meeting progresses on two topics, "proposing a new project" and "checking the budget," the server notifies the user of the summary. This allows users to always understand the progress of the meeting.

[0774] Next, after the meeting ends, the server automatically generates minutes based on the text data of comments and facial expression and emotion analysis data. For example, detailed minutes such as "Speaker A: Regarding the new project proposal" and "Speaker B: Opinions on the proposal" are created. Furthermore, each speaker is scored based on the number and content of their comments, taking into account the quality of their comments and their emotional state. Users can check the number of comments they made and the number of important opinions they expressed, and use this as an indicator to improve meeting performance.

[0775] Below are some examples of prompts to input to a generative AI model:

[0776] 1. When silence occurs during a meeting, notifications are sent to participants who are not concentrating or showing negative emotions to encourage them to speak up.

[0777] 2. As the meeting progresses, a summary of what was discussed is sent every 15 minutes to all participants' devices.

[0778] 3. After the meeting, minutes are automatically generated based on the content of the comments and the analysis data, and each speaker is scored, taking into account the results of the emotion engine.

[0779] In this way, the system facilitates smooth meeting progress and supports effective discussions that take into account the emotional state of participants.

[0780] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0781] Step 1:

[0782] Starting a meeting and capturing data

[0783] When a user presses the conference start button on the terminal, the terminal starts capturing the conference video and audio streams. These streams are sent to the server as they are. The input is video and audio data from the camera and microphone connected to the terminal, and the output is sent to the server as stream data in real time.

[0784] Specifically, when a user presses a button to join a conference, the camera captures video and the microphone captures audio, and then these are sent to the server.

[0785] Step 2:

[0786] Video and audio stream analysis

[0787] The server analyzes the received video and audio streams using a generative AI model, converting speech into text using speech recognition technology (input: audio stream, output: text data), and analyzing facial expressions and movements using facial and movement recognition algorithms (input: video stream, output: facial and movement data).

[0788] Specifically, when a user says, "What do you think about this proposal?", the voice data is converted into text and saved as text data saying, "What do you think about this proposal?" At the same time, the user's facial expressions and movements are extracted from the video stream and converted into data.

[0789] Step 3:

[0790] Emotional state analysis using emotion engine

[0791] The server uses an emotion engine to determine the participant's emotional state from the analyzed facial expressions and tone of voice. The emotion engine categorizes the emotion from the analysis results (input: facial expression and tone of voice data, output: emotional state data) and uses that data for the next step.

[0792] Specifically, the emotion engine analyzes the facial expressions and tone of voice of the user when they speak, and converts emotions such as "joy," "anger," and "surprise" into data.

[0793] Step 4:

[0794] Silence detection and prompting

[0795] The server monitors the audio data during the conference and detects silence for a certain period of time (for example, 10 seconds) (input: audio data, output: silence detection status). When silence is detected, participants with negative emotional states or participants with declining concentration are identified based on the analysis results of the emotion engine, and a notification is sent to those participants encouraging them to speak (input: emotional state data, output: notification encouraging them to speak).

[0796] Specifically, the server detects 10 seconds of silence and sends a notification to participants whose emotion engine indicates a negative emotional state, saying, "Please give your opinion next."

[0797] Step 5:

[0798] Meeting summary and distribution

[0799] The server summarizes the discussion every 15 minutes. Using a generative AI model, it extracts key topics and decisions from each comment and creates a summary (input: comment data, output: summary data). This is then sent to all participants' devices.

[0800] Specifically, the server summarizes the contents of the 15-minute meeting into the topics "proposing a new project" and "confirming the budget," and distributes the summary information to the terminals of all participants.

[0801] Step 6:

[0802] Automatic generation and scoring of meeting minutes after the meeting

[0803] After the meeting ends, the server automatically generates minutes based on the accumulated remarks and analysis data. Using a generative AI model, the server records the speaker's name, remarks, timing of remarks, and emotional state in detail (input: remarks and analysis data, output: minutes data). Furthermore, it scores each speaker based on the number and content of their remarks (input: remarks data, output: scoring data). This is then sent to each participant's device.

[0804] Specifically, the server creates detailed minutes such as "Speaker A: Regarding the proposal for a new project" and "Speaker B: Comments on the proposal," and then notifies each participant of the scoring results, such as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments."

[0805] (Application example 2)

[0806] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] The issues of declining work efficiency and ensuring worker safety in factories are extremely important. Conventional systems have difficulty monitoring and analyzing the progress of work, the concentration level of workers, and their emotional state in real time, and issuing appropriate instructions or warnings as needed. Another issue is that while efforts to improve work efficiency have been made, there has been a lack of efforts to promote breaks that take into account workers' fatigue and emotional state.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0809] In this invention, the server includes: means for receiving video and audio streams during a conference; means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; means for sending notifications to encourage participants to speak; means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion; means for sending the summarized conference summaries to all participants' terminals; means for automatically generating minutes after the conference based on all comments and analysis data; means for scoring each participant based on the number of comments and the content of their comments; means for individually sending the scored results based on the number of comments and the content of their comments; means for monitoring work in real time and issuing a warning when a specific movement is detected; means for analyzing the concentration and emotional state of workers and encouraging them to take a break as necessary; and means for instructing the next work step when efficiency drops, thereby improving work efficiency in factories and ensuring worker safety.

[0810] A "video stream" refers to video data that is continuously sent from a device such as a camera.

[0811] An "audio stream" refers to a continuous stream of audio data sent from a microphone or other audio input device.

[0812] The term "analysis means" refers to a method or device for analyzing the content of speech, facial expressions, and actions of participants based on the video stream and audio stream.

[0813] "Means for detecting silence" refers to a method or device for detecting a period of no speech.

[0814] "Notification means" refers to a method or device for notifying participants of analysis results, warnings, instructions, etc.

[0815] "Means for summarizing the meeting" refers to a method or device for periodically summarizing what has been said and the progress of discussions during a meeting.

[0816] "Means for automatically generating minutes" refers to a method or device for creating minutes after a meeting based on the content of remarks and analysis data.

[0817] "Scoring means" refers to a method or device for evaluating each participant based on the number of comments and the content of their comments.

[0818] "Real-time monitoring means" refers to a method or device for monitoring and analyzing the real-time progress of work.

[0819] "Alert means" refers to a method or device for issuing an alert when a particular action or condition is detected.

[0820] "Means to encourage breaks" refers to methods or devices for analyzing a worker's concentration level and emotional state and instructing them to take a break as necessary.

[0821] "Means for indicating work steps" refers to a method or device for indicating the next work procedure when work efficiency drops.

[0822] The present invention relates to a work supervision system that aims to improve work efficiency in a factory and ensure the safety of workers. A specific embodiment of this system will be described below.

[0823] System configuration

[0824] Hardware configuration:

[0825] Smart glasses: Devices worn by workers that capture visual and audio information.

[0826] Server: A high-performance computer that analyzes collected data and generates instructions and warnings.

[0827] Camera: A device installed in each work area in a factory that provides a video stream.

[0828] Microphone: A device used to capture the voice of workers.

[0829] Software configuration:

[0830] Generative AI model (e.g., GPT-3): Converts input voice data into text and analyzes what is being said.

[0831] Emotion analysis engine (e.g., Emotion API): Analyzes facial expressions and movements from input video data to determine emotional state.

[0832] Streaming processing software (e.g., WebRTC): Sends and receives video and audio streams in real time.

[0833] Program processing

[0834] The server receives video and audio streams from the smart glasses and camera. The received data is analyzed in real time to analyze the worker's speech, facial expressions, and movements. This analysis is performed using a generative AI model and an emotion analysis engine. Based on the analysis results, the following processes are performed:

[0835] 1. Real-time monitoring: There is a way to monitor the progress of work and issue alerts when certain conditions are met (e.g., not wearing proper protective equipment).

[0836] 2. Promoting breaks: Analyze workers' concentration levels and emotional states and encourage them to take breaks as needed.

[0837] 3. Efficiency: There is a way to indicate the next work step if work efficiency declines.

[0838] Specific examples

[0839] Example 1: When worker A's concentration level is low

[0840] Based on the results of the server's analysis, the smart glasses will notify the user, "Your concentration level is declining. Please take a break."

[0841] Example 2: When worker B is not wearing protective equipment

[0842] The smart glasses notify workers to wear protective equipment, encouraging them to work safely.

[0843] Prompt Sentence Examples

[0844] Example prompt for generative AI model:

[0845] "Design an AI system to manage the work of the next shift. To ensure worker safety and efficiency, the system should include the following elements: facial expression analysis, voice recognition, emotion analysis, and real-time warning instructions. The following are specific prompts for analysis. The proposed configuration includes: when a worker is not wearing protective gear, when work efficiency drops sharply, how to issue appropriate instructions, how to communicate data between the server and the smart glasses, and what to display on the smart glasses."

[0846] The above is a specific embodiment of the present invention. This system realizes improved efficiency and safety in factory work.

[0847] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0848] Step 1:

[0849] The smart glasses and camera capture video and audio streams and send them to a server.

[0850] Input: Video and audio streams from smart glasses and camera.

[0851] Output: Sending video and audio streams to the server.

[0852] Step 2:

[0853] The server analyzes the received video and audio streams using a generative AI model and emotion analysis engine.

[0854] Input: Video and audio streams.

[0855] Output: Analysis data on each participant's speech, facial expressions, and movements.

[0856] Specific operations: Converting voice data into text and analyzing spoken content using a generative AI model, and analyzing facial expressions and movements using an emotion analysis engine.

[0857] Step 3:

[0858] The server monitors the progress of work based on the analysis results and detects specific actions or conditions.

[0859] Input: Analysis data.

[0860] Output: The result of detecting a specific behavior or state.

[0861] Specific actions: Checking that workers are wearing the correct protective equipment, detecting facial expressions such as fatigue or impatience.

[0862] Step 4:

[0863] The server sends an alert notification to the smart glasses when certain conditions are met.

[0864] Input: The detection of a specific action or state.

[0865] Output: Alert notification to smart glasses.

[0866] Specific operation: If a worker is not wearing protective equipment, the system notifies the worker to "put on protective equipment."

[0867] Step 5:

[0868] The server analyzes the worker's concentration level and emotional state and encourages them to take a break if necessary.

[0869] Input: Analysis data.

[0870] Output: Break promotion notification to smart glasses.

[0871] Specific behavior: If a worker's concentration level is decreasing, the system will notify them, "Your concentration level is decreasing. Please take a break."

[0872] Step 6:

[0873] The server instructs the next work step when the work efficiency drops.

[0874] Input: Data about work progress and efficiency.

[0875] Output: Next work step instructions to the smart glasses.

[0876] Specific behavior: If work is stalled, notify the user to "proceed to the next work step."

[0877] Through the above steps, efficiency and safety of work in the factory are improved.

[0878] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0879] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0880] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0881] [Third embodiment]

[0882] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0883] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0884] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0885] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0886] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0887] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0888] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0889] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0890] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0891] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0892] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0893] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0894] The system of the present invention is designed to analyze the video and audio streams of a conference in real time and facilitate the progress of the conference. A specific embodiment of this system will be described below.

[0895] The server receives the video and audio streams of the conference. It then analyzes the received video and audio streams to analyze the participants' speech, facial expressions, and movements in real time. Based on the analysis results, the server detects silence and identifies participants who should be encouraged to speak. It also sends notifications to encourage them to speak.

[0896] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The summarized meeting summary is sent to the devices of all participants. After the meeting ends, minutes are automatically generated based on all comments and analysis data. Each participant is scored based on the number of comments and the content of their comments, and the results are sent individually.

[0897] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[0898] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[0899] The operation of the system will be explained below using specific examples.

[0900] Example 1: Silence detection and prompting

[0901] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be prompted to speak next. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who then gives their opinion based on it.

[0902] Example 2: Meeting summary and sharing

[0903] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[0904] Example 3: Minutes and scoring

[0905] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[0906] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[0907] The processing flow will be explained below.

[0908] Step 1:

[0909] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[0910] Step 2:

[0911] The server analyzes the received video and audio streams. Specifically, it uses a generative AI model to analyze participants' speech, facial expressions, and movements in real time. Speech is converted into text, and facial expressions and movements are digitized as analysis results.

[0912] Step 3:

[0913] The server detects silence during the conference based on the analysis results. If no speech is made for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings.

[0914] Step 4:

[0915] When the server detects silence, it prompts the most appropriate participant to speak. It identifies participants who are not concentrating or appear sleepy from the analysis results and sends them a notification to prompt them to speak.

[0916] Step 5:

[0917] The device displays the notification received from the server to the user. Specifically, it displays a message such as "Please state your opinion next" on the screen to encourage the user to speak.

[0918] Step 6:

[0919] The user checks the notification displayed on the device and follows the instructions to express their opinions or ask questions. The comments are then sent back to the server via the device.

[0920] Step 7:

[0921] The server summarizes the meeting at regular intervals (for example, 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. This summary is generated in text format.

[0922] Step 8:

[0923] The server sends a summary of the meeting to all participants' terminals, which receive the summary message and display it to the user.

[0924] Step 9:

[0925] As the meeting progresses, the server continuously records what is said and the analytical data, so that all comments and discussions are stored in a log.

[0926] Step 10:

[0927] After the meeting, the server automatically generates minutes based on the accumulated remarks and analysis data. Specifically, it organizes the remarks made by each speaker and important points of discussion.

[0928] Step 11:

[0929] The server scores each participant based on the number and content of their comments. For example, the evaluation score is calculated based on multiple indicators such as the number of comments, the quality of the comments, and the degree of contribution to the discussion.

[0930] Step 12:

[0931] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[0932] Step 13:

[0933] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement.

[0934] The above are the specific processing steps of the present system based on the claims. This system allows meetings to proceed smoothly and enables highly productive discussions.

[0935] Example 1

[0936] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] Conventional conference systems lack sufficient support functions to facilitate smooth meeting progress, making it difficult for all participants to participate effectively in discussions. Furthermore, recording comments, generating minutes, and analyzing the content of comments must be done manually, which requires a great deal of time and effort. Another issue is that there are few ways to grasp participants' own contributions to the discussion, making it difficult to obtain specific feedback to improve the quality and efficiency of the meeting.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0939] In this invention, the server includes means for receiving video data and audio data during a conference, means for analyzing the received video data and audio data and analyzing the participants' comments, facial expressions, and movements in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion, means for transmitting the summarized conference summaries to all participants' terminals, means for automatically generating minutes after the conference based on all comments and the analysis data, means for scoring each participant based on the number of comments and the content of their comments, means for individually transmitting the scoring results based on the number of comments and the content of their comments, means for using computer vision technology to analyze the video data, means for using voice recognition technology to analyze the audio data, means for monitoring the audio level to detect silence for a certain period of time, means for notifying all participants of the generated conference summary, and means for distributing the minutes and scoring results to all participants.This allows the conference to proceed smoothly and allows all participants to participate efficiently in the discussion. In addition, automated recording and analysis of meetings reduces effort and time, and allows each participant to understand their own contribution and receive specific feedback that can help improve their meeting skills.

[0940] "Video data during a meeting" refers to visual information acquired through a camera while the meeting is in progress, and includes visual content such as the movements and facial expressions of participants.

[0941] "Audio data" refers to acoustic information acquired through a microphone during a conference, including speeches made by participants and other audio information.

[0942] "Means for receiving" refers to devices or software that provide the function of acquiring video data and audio data and inputting them into the server.

[0943] "Means for analyzing" refers to an algorithm or computer program that processes the received data and extracts specific information.

[0944] "Real-time analysis" refers to the act of processing acquired data immediately to obtain results, providing information immediately without delay.

[0945] "Silence detection means" means a device or software that provides the functionality to identify and detect periods of silence within audio data.

[0946] "Means for identifying participants who should be encouraged to speak" refers to algorithms or computer programs that use the analyzed data to identify the next person to speak.

[0947] "Means for sending notifications" refers to machines and programs for sending messages or alerts to specific participants.

[0948] A "meeting summary tool" refers to an algorithm or computer program that concisely summarizes what was discussed during a meeting.

[0949] "Means for sending a conference summary to all participants' terminals" refers to machines and programs that distribute messages to convey summarized information to all participants.

[0950] "Means for automatically generating minutes" refers to algorithms or computer programs that automatically create detailed records of meetings based on what was said during the meeting and analytical data.

[0951] "Means for scoring" refers to an algorithm or computer program that evaluates the number and content of each participant's comments and calculates a score based on that.

[0952] "Means for individual transmission" refers to a machine and program that provides the functionality for transmitting scoring results to each participant's terminal in a specific format.

[0953] "Computer vision technology" refers to a collection of techniques and algorithms used to extract information from digital images and video.

[0954] "Speech recognition technology" refers to a collection of techniques and algorithms for analyzing voice data and converting it into text.

[0955] "Means for monitoring audio levels" refers to an algorithm or computer program that continuously monitors the volume of audio data and detects fluctuations in the volume.

[0956] The system of the present invention is designed to analyze the video and audio streams of a conference in real time to facilitate the progress of the conference. A specific embodiment of this system will be described below, with each role played by a server, a terminal, and a user.

[0957] Server Roles

[0958] The server processes data and provides functions to support the conference through the following procedures:

[0959] 1. Receiving data:

[0960] The server receives video and audio data during the conference using a communication protocol such as WebRTC, which transmits real-time video and audio streams from each participant to the server.

[0961] 2. Video and audio data analysis:

[0962] The received video data is analyzed using OpenCV and other computer vision technologies, specifically using facial recognition and expression analysis algorithms to understand the participants' facial expressions and movements.

[0963] The voice data is converted into text using voice recognition technology (for example, Google Cloud Speech-to-Text API), which makes the spoken content available as text data.

[0964] 3. Silence detection and prompting:

[0965] The server continuously monitors the audio level and detects silence for a certain period (for example, 10 seconds) using a specific algorithm that monitors the audio level.

[0966] When silence is detected, an algorithm determines who should be the next person to speak and sends them a notification, such as "You're next."

[0967] 4. Summary of the meeting:

[0968] Based on what was said during the meeting and the progress of the discussion, a generative AI model is used to summarize the main topics and decisions of the meeting at regular intervals (for example, every 15 minutes).

[0969] A summary of the meeting is sent to all participants' devices, making it easier for them to understand the flow of the meeting.

[0970] 5. Generate transcripts:

[0971] After the meeting, minutes are automatically generated based on all statements and analysis data. The generated minutes include a detailed record of what each speaker said.

[0972] 6. Scoring:

[0973] Each participant is scored based on the number of comments and the content of their comments. This scoring is based on the evaluation criteria of the frequency, importance, and contribution of comments.

[0974] The scored results are sent individually to each participant's device.

[0975] Device Role

[0976] The terminal provides the user with visual and auditory information about the data sent from the server, and transmits user operations to the server.

[0977] 1. Displaying data:

[0978] The terminal displays the video and audio streams sent from the server, allowing users to view the contents of the conference in real time.

[0979] 2. Capturing what users say and do:

[0980] When a user speaks, the device captures the user's voice and sends it to the server. It also captures the user's facial expressions and movements and sends this data to the server for analysis.

[0981] 3. Notification and Summary Display:

[0982] When a notification encouraging speaking or a summary of the conference is sent from the server, the terminal displays this to the user as a pop-up notification or message.

[0983] User Roles

[0984] Users can participate in the conference and make comments and hold discussions as usual. In addition, they can check notifications and summaries from the server and use them to express their opinions and advance the discussion.

[0985] 1. Statement:

[0986] The user speaks through a microphone, and the content of the speech is transmitted to the server via the terminal.

[0987] Facial expressions and movements are also captured by the camera and sent to the server.

[0988] 2. Responding to Notices:

[0989] When a user receives a notification from the server prompting the user to speak, the user responds by speaking.

[0990] 3. Review of minutes and scoring results:

[0991] After the meeting, users can check the minutes and scoring results displayed on their device to help improve their own meeting skills.

[0992] Specific examples

[0993] Example 1: Silence detection and prompting

[0994] During a meeting, if a participant finishes speaking and there is a 10-second silence, the server detects the silence and identifies the next participant to be prompted to speak. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who can then express their opinion.

[0995] Example 2: Meeting Summary

[0996] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[0997] Example 3: Minutes and scoring

[0998] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[0999] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[1000] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1001] Step 1:

[1002] Receiving data

[1003] The server receives video and audio data during the conference using the WebRTC protocol.

[1004] Input: Real-time video and audio streams sent from each participant's device.

[1005] Data processing: Video and audio data are received in the form of packets.

[1006] Output: Received video and audio data.

[1007] How it works: The server uses the WebRTC protocol to receive video and audio streams from each participant in real time, store them in a database, and queue them for processing.

[1008] Step 2:

[1009] Video data analysis

[1010] The server analyzes the received video data using the OpenCV library.

[1011] Input: Video data received via the WebRTC protocol.

[1012] Data processing: Applying facial recognition and expression analysis algorithms to extract specific facial features and changes in facial expressions.

[1013] Output: Participant's facial expression and movement data.

[1014] Specific operation: The server uses OpenCV to detect facial features for each video frame and analyzes changes in facial expression and eye movements, thereby classifying facial expressions into categories such as "smiling," "serious," and "sleepy."

[1015] Step 3:

[1016] Analysis of audio data

[1017] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[1018] Input: Audio data received via the WebRTC protocol.

[1019] Data processing: Using voice recognition technology, voice data is converted into text data.

[1020] Output: Text data of the speech.

[1021] What it does: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts what is said into text in real time, which is then stored in a database for further analysis and recording.

[1022] Step 4:

[1023] Silence Detection

[1024] The server monitors the analyzed audio data and detects periods of silence.

[1025] Input: Speech and speech level data converted to text.

[1026] Data processing: Based on the voice level data, monitor whether there is any continuous speech for a specific period (for example, 10 seconds).

[1027] Output: Silence detection results.

[1028] What happens: The server checks that the audio level is below a certain threshold and then runs an algorithm that detects silence if no speech is heard for 10 consecutive seconds.

[1029] Step 5:

[1030] Sending a call to speak

[1031] If a period of silence continues, the server identifies the participant who should next be prompted to speak and sends a notification.

[1032] Input: Silence detection results and analyzed facial expression data.

[1033] Data processing: Based on the analysis results obtained from video and audio data, participants who should be encouraged to speak are identified.

[1034] Output: Informational message.

[1035] Specific operation: The server sends a notification to the identified participant's device saying, "Please give your opinion next." This notification is displayed as a pop-up message on the device.

[1036] Step 6:

[1037] Conference Summary

[1038] The server periodically summarizes the contents of the conference.

[1039] Input: Text data converted from audio data and discussion progress.

[1040] Data processing: Using NLP techniques, we analyze what is being said and extract key topics and decisions.

[1041] Output: Meeting summary text.

[1042] Specific operation: The server uses a generative AI model to summarize the contents of the meeting every 15 minutes and sends the summary text to the devices of all participants.

[1043] Step 7:

[1044] Generate meeting minutes

[1045] After the meeting ends, the server automatically generates minutes based on all statements and analysis data.

[1046] Input: Text data and analysis data of all statements.

[1047] Data processing: Integrate the statements and analytical data and organize them into detailed records.

[1048] Output: Auto-generated meeting transcript.

[1049] Specific operation: The server automatically generates minutes based on all data collected during the meeting and sends them to each participant's device. The minutes include the remarks and decisions made by each participant.

[1050] Step 8:

[1051] Sending scoring results

[1052] The server scores each participant based on the number of comments and the content of their comments, and sends the results individually.

[1053] Input: All speech content and number of speeches data.

[1054] Data processing: A score is calculated for each speaker based on the number of comments, importance, and contribution of each speaker.

[1055] Output: Scoring results.

[1056] How it works: The server evaluates and scores each participant's performance, taking into account the frequency of their comments and the importance of their content. The results are sent to each participant's device as individual notifications.

[1057] These are the specific processing steps of the system. This allows meetings to proceed smoothly and enables all participants to participate in the discussion efficiently. In addition, the automatic generation of meeting minutes and feedback of scoring results can contribute to improving the quality of meetings and the skills of participants.

[1058] (Application example 1)

[1059] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1060] Conventional conference systems have difficulty efficiently and effectively managing tasks such as encouraging participants to speak and creating meeting minutes. In particular, current conference management systems are insufficient in production sites such as factories, where rapid decision-making and communication are required. Furthermore, there is no system for analyzing participants' facial expressions and movements in real time, making it difficult to properly grasp the status of participants during the meeting, which can slow down the progress of the meeting. Therefore, an effective system to solve these issues is needed.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1062] In this invention, the server includes means for receiving video and audio streams during a conference, means for analyzing the received video and audio streams and analyzing the speech content, facial expressions, and movements of participants in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for analyzing the video and audio streams and encouraging specific participants to speak to support in-factory conferences, and means for applying a facial expression analysis model using the video and audio streams, identifying facial expressions based on the analysis results, and determining participants who should be encouraged to speak, thereby enabling quick decision-making and efficient communication within the factory.

[1063] A "video stream" is a series of video data captured during a conference via a video input device such as a camera.

[1064] An "audio stream" is a series of audio data captured during a conference via an audio input device such as a microphone.

[1065] "Real-time analysis" refers to the process of analyzing captured video and audio streams in real time and providing results without delay.

[1066] "Silence detection" refers to determining whether a silence state has occurred for a certain period of time or longer based on the analyzed audio stream.

[1067] "Speaker prompt notification" is a function that sends a message to specific participants to prompt them to speak when silence is detected.

[1068] The "facial expression analysis model" is a machine learning model for analyzing participants' facial expressions obtained from video streams and estimating their emotional states.

[1069] The "analysis results" are information on speech content, facial expressions, and movements obtained from the video stream and audio stream data.

[1070] "Scoring" is the process of evaluating each participant based on the number and content of their comments.

[1071] Minutes are documents that record what was said and what was decided at a meeting.

[1072] "Factory meetings" are meetings held in factories for the purpose of planning and solving problems related to production.

[1073] "Notification means" is a function for sending messages and instructions from the system to participants.

[1074] "Multimodal analysis" is the process of simultaneously analyzing multiple different data sources (such as video and audio streams) and utilizing the results in a comprehensive manner.

[1075] The present invention provides a system for smoothly conducting meetings in a factory and promoting efficient decision-making. Specific embodiments for implementing this system will be described below.

[1076] System Overview

[1077] This system consists of three roles: a server, a terminal, and a user. The server plays a central role in receiving and analyzing the video and audio streams of the conference. The terminal is a device through which participants interact with the system, displaying the video and audio streams and receiving notifications. The user participates in the conference, expressing their opinions and participating in discussions.

[1078] Hardware and Software Configuration

[1079] Hardware: Camera, microphone, display, smartphone.

[1080] Software: Python, OpenCV (video stream processing), TensorFlow (facial expression analysis model), Numpy (data processing).

[1081] Data processing and calculation

[1082] Server Processing

[1083] 1. Receiving video and audio streams

[1084] The server receives the video and audio streams from the camera and microphone in real time.

[1085] 2. Real-time analysis

[1086] We use OpenCV to detect participants' faces from the video stream, TensorFlow to analyze detected facial expressions and estimate participants' emotional states, and Numpy to analyze the volume levels of the audio stream and detect silence.

[1087] 3. Detect silence and send notifications

[1088] If silence continues for a certain period of time, the system will identify the participant who should be prompted to speak next based on the analysis results and send a notification to that participant, which will be displayed on their device.

[1089] 4. Meeting summary and minutes generation

[1090] Based on the main comments made during the meeting and the progress of the discussion, a summary of the meeting is generated at regular intervals (for example, every 15 minutes) and sent to the devices of all participants. After the meeting, minutes are automatically generated based on all comments and analysis data, and the number and content of comments made by each participant are scored.

[1091] Terminal handling

[1092] 1. Displaying video and audio streams

[1093] The terminal displays the video and audio streams sent from the server, allowing participants to visually understand the contents of the conference.

[1094] 2. Receiving and Displaying Notifications

[1095] It receives notifications from the server and displays them to users, who can then express their opinions and advance the discussion.

[1096] Specific examples

[1097] If there is silence during a meeting, the system will:

[1098] 1. At the beginning of the meeting, the statement is made, "We will begin discussing the design of the new product."

[1099] 2. If there is a period of silence, the server detects this and identifies the next participant to speak.

[1100] 3. A notification will appear on the device of the person you are speaking to saying, "Yamada-san, please comment on the next steps."

[1101] 4. This allows Yamada to express his opinion and the meeting proceeds smoothly.

[1102] An example prompt is:

[1103] Meeting start: "We're going to start discussing the design of our new product."

[1104] There is silence for a while.

[1105] Prompt to speak: "Yamada-san, please comment on next steps."

[1106] In this way, this system enables smooth progress in factory meetings and efficient decision-making.

[1107] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1108] Step 1:

[1109] Receiving video and audio streams

[1110] The server receives video and audio streams in real time through cameras and microphones. The input is the video and audio data of the meeting, and the output is the raw data stored and prepared for analysis. In this step, the server continuously acquires data by capturing the streams from each device.

[1111] Step 2:

[1112] Real-time analysis of video streams

[1113] The server uses OpenCV to detect participants' faces from the video stream. The input is the received video stream data, and the output is the location information of the detected faces. This face location information becomes the input for subsequent facial expression analysis. Specifically, the server recognizes the contours of the face from the video frame and identifies its location.

[1114] Step 3:

[1115] Application of facial expression analysis model

[1116] The server performs facial expression analysis on the detected faces using TensorFlow. The input is image data of salient faces and a pre-trained facial expression analysis model, and the output is an estimated emotional state (e.g., joy, anger, sadness, happiness, etc.). This is the process of applying the facial expression analysis model to classify facial expressions.

[1117] Step 4:

[1118] Parsing an Audio Stream

[1119] The server uses Numpy to analyze the volume level of the audio stream and detect silence. The input is the audio stream data acquired in real time, and the output is the volume level (silence or not) judgment result. Specifically, it calculates the amplitude of the audio data and determines silence if it is below a preset threshold.

[1120] Step 5:

[1121] Detect silence and send prompts

[1122] If silence continues for a certain period of time (e.g., 10 seconds), the server identifies the participant who should be prompted to speak next based on the results of facial expression analysis. The input is the silence detection result and the results of facial expression analysis, and the output is the identified participant ID and the corresponding prompt to speak. The server then sends a "please speak next" notification to this identified participant.

[1123] Step 6:

[1124] Viewing video and audio streams

[1125] The terminal receives the video and audio streams sent from the server and displays them to the user. The input is the video and audio stream data from the server, and the output is the display of that data on the terminal. This allows the user to grasp the status of the conference in real time.

[1126] Step 7:

[1127] Receiving and Viewing Notifications

[1128] The terminal receives a prompt to speak from the server and displays it to the user. The input is the prompt to speak notification data from the server, and the output is the display of the notification to the user. Upon receiving this notification, the user can express their opinion or advance the discussion.

[1129] Step 8:

[1130] Generate meeting summaries and minutes

[1131] The server generates a summary of the meeting at regular intervals (e.g., every 15 minutes) based on the main comments made during the meeting and the progress of the discussion, and sends it to the devices of all participants. After the meeting ends, it automatically generates minutes based on all comments and analysis data, scoring the number and content of comments made by each participant. The input is all comment data and analysis results from the meeting, and the output is summary information and minutes. Specifically, it uses natural language processing technology to create summaries, extract necessary information, and document it.

[1132] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1133] The present invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. Specific embodiments of this system are described below.

[1134] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows participants' speech, facial expressions, actions, and emotions to be analyzed in real time. Speech is converted into text, and facial expressions and actions are converted into data as analysis results. The emotion engine also recognizes participants' emotional states from their facial expressions and tone of voice.

[1135] Based on the analysis results, the server detects silence during the meeting. If no one speaks for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings. When silence is detected, the server also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. For example, it sends a notification encouraging participants who are determined to be not concentrating or whose emotional state is changing negatively to speak.

[1136] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[1137] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[1138] The operation of the system will be explained below using specific examples.

[1139] Example 1: Silence detection and prompting

[1140] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be encouraged to speak next. If the emotion engine's analysis determines that a particular participant is losing concentration or showing negative emotions, it sends that participant a notification saying, "Please speak next." The device displays this notification to the user, who then expresses their opinion based on it.

[1141] Example 2: Meeting summary and sharing

[1142] As the meeting progresses, the server summarizes the content of the discussion every 15 minutes. For example, it briefly summarizes the main topics and decisions discussed in the first 15 minutes and sends this content to the devices of all participants. The analysis results of the emotion engine are also taken into account, and summary information corresponding to increases in negative emotions and decreases in positive emotions is provided. The device displays this summary information to the user, allowing them to always understand the progress of the meeting.

[1143] Example 3: Minutes and scoring

[1144] After the meeting ends, the server automatically generates minutes based on the accumulated content of comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding a new project proposal" and "Speaker B: Opinions on the proposal." The analysis results of the emotion engine are also reflected, and the emotional state of each speaker is included in the minutes. In addition, each speaker is scored based on the number of comments and content of their comments. For example, they might be evaluated as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments." The scoring also takes into account the quality of the comments and the emotional state. These results are sent individually to each participant's device, allowing users to check their own performance.

[1145] The above is a specific embodiment of the present invention. This system not only allows meetings to proceed smoothly, but also allows for more effective discussions by taking into account the emotional states of participants. The introduction of an emotion engine will improve the quality of meetings and also help improve individual meeting skills.

[1146] The processing flow will be explained below.

[1147] Step 1:

[1148] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[1149] Step 2:

[1150] The server uses generative AI models and emotion engines to analyze the incoming video and audio streams in real time, converting participants' speech into text and digitizing their facial expressions, movements, and emotional states.

[1151] Step 3:

[1152] Based on the analysis results, the server monitors speech content, facial expressions, and movements to detect silence. If no speech is heard for a certain period of time (for example, 10 seconds), the server determines this as silence.

[1153] Step 4:

[1154] When the server detects silence, it also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. If a specific participant is losing concentration or showing negative emotions, it sends a notification encouraging them to speak. At this time, an appropriate message is selected depending on the situation.

[1155] Step 5:

[1156] The terminal displays the notification received from the server to the user. Specifically, a message such as "Please give your opinion next" is displayed on the screen of the user terminal.

[1157] Step 6:

[1158] Users can check the notifications displayed on their devices and express their opinions or questions. The content of their comments is captured by the device and sent to the server in real time.

[1159] Step 7:

[1160] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The server also reflects the analysis results of the emotion engine, and generates summary information that takes into account changes in emotions.

[1161] Step 8:

[1162] The server sends a summary of the conference to all participants' terminals, which receive the summary information and display it to the user, allowing the user to keep track of the conference progress.

[1163] Step 9:

[1164] The server records the content of comments and analysis data that is accumulated in real time, and stores the data necessary for creating minutes after the meeting.

[1165] Step 10:

[1166] After the meeting, the server automatically generates minutes based on the saved remarks and analysis data, including the remarks made by each speaker, key discussion points, and emotional state.

[1167] Step 11:

[1168] The server scores each participant based on the number and content of their comments, which reflects the quality, quantity, and emotional state of their comments.

[1169] Step 12:

[1170] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[1171] Step 13:

[1172] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement in the future.

[1173] These are the specific processing steps of this system, which combines an emotion engine. This system ensures smooth progress in meetings and highly productive discussions. Furthermore, by taking into account the emotional state of participants, the quality of meetings can be further improved.

[1174] Example 2

[1175] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] Conventional conferencing systems lack the means to effectively analyze participants' remarks, facial expressions, actions, and emotions in real time, resulting in issues such as the inability to smoothly progress through the meeting and the inability to respond to changes in participants' concentration and emotional state. Furthermore, there are insufficient means to prompt appropriate participants to speak when silence persists, resulting in a decrease in meeting efficiency. Furthermore, summarizing discussions during meetings and creating meeting minutes after the meeting are often done manually, which is time-consuming and labor-intensive.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1178] In this invention, the server includes: a means for receiving video and audio streams during a conference; a means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; an emotion engine including means for determining participants' emotional states from facial expressions and tone of voice; a means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; a means for sending notifications encouraging participants who show a negative emotional state or a lack of concentration to speak; a means for summarizing the conference at regular intervals based on the content of comments made during the conference and the progress of the discussion; a means for sending the summarized conference summary to all participants' terminals; a means for automatically generating minutes after the conference based on all comments and analysis data; a means for scoring each participant based on the number of comments and the content of their comments; and a means for individually transmitting the scored results. This not only facilitates smooth progress in the conference, but also takes into account the participants' emotional states, enabling more effective discussions. Furthermore, the creation of summaries during the conference and the automatic generation of minutes after the conference ends significantly reduces time and effort.

[1179] 1. "Video Stream" means a data stream for transmitting and receiving video of participants in real time during a conference.

[1180] 2. "Audio Stream" means a data stream for transmitting and receiving the voices of participants in a conference in real time.

[1181] 3. "Analysis" refers to the process of decomposing and processing the content of received video and audio streams to extract various information.

[1182] 4. "Speech content" refers to the text information spoken by each participant during the meeting.

[1183] 5. “Facial expression” refers to information about a participant’s emotional state that can be read from their facial movements and expressions.

[1184] 6. "Movement" refers to information about participants' body movements and gestures.

[1185] 7. "Emotional state" refers to the psychological state of the participant as analyzed based on facial expressions and tone of voice.

[1186] 8. The "Emotion Engine" is software that analyzes and determines participants' emotional state from facial expressions and tone of voice.

[1187] 9. "Silence detection" is the process of automatically detecting periods of inactivity during a meeting.

[1188] 10. A "prompt to speak" is a message sent to prompt a specific participant to speak next when there is a period of silence.

[1189] 11. A "meeting summary" is a concise summary of the major topics and decisions discussed during the meeting.

[1190] 12. "Summarizing" refers to extracting important points based on what was said during a meeting and the progress of the discussion, and summarizing them in a concise form.

[1191] 13. "Minutes" means a document that contains a detailed record of what was discussed, what decisions were made, and what each speaker said at a meeting.

[1192] 14. "Scoring" is the process of evaluating each participant based on the number of comments and the quality of their comments.

[1193] 15. "Multimodal analysis" refers to the technique of simultaneously analyzing multiple data formats (e.g., audio and video).

[1194] 16. "Timer means" means a device or software that measures a certain period of time and monitors whether or not any speech is made during that period.

[1195] This invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. A specific embodiment of this system is described below.

[1196] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows for real-time analysis of participants' speech, facial expressions, actions, and emotions. Specifically, speech is converted into text using speech recognition technology. Facial and motion recognition algorithms are used to analyze facial expressions and actions. The emotion engine determines participants' emotional states from their facial expressions and tone of voice. Based on the results of this analysis, the server detects silence and prompts participants who show negative emotional states or low concentration to speak.

[1197] For example, if a 10-second silence occurs during a meeting, participants who are showing signs of low concentration or negative emotions can be notified by the emotion engine's analysis, with the message "Please speak next." The device that receives this notification will display it to the user, encouraging them to speak next.

[1198] The device displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the device captures the content and sends it to the server. It also captures the user's facial expressions and movements and sends them to the server. Furthermore, if the server sends a notification encouraging the user to speak or a summary of the conference, the device displays it to the user.

[1199] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[1200] For example, the server summarizes the meeting every 15 minutes based on what is said during the meeting and the progress of the discussion, and sends this summary to the devices of all participants. If the meeting progresses on two topics, "proposing a new project" and "checking the budget," the server notifies the user of the summary. This allows users to always understand the progress of the meeting.

[1201] Next, after the meeting ends, the server automatically generates minutes based on the text data of comments and facial expression and emotion analysis data. For example, detailed minutes such as "Speaker A: Regarding the new project proposal" and "Speaker B: Opinions on the proposal" are created. Furthermore, each speaker is scored based on the number and content of their comments, taking into account the quality of their comments and their emotional state. Users can check the number of comments they made and the number of important opinions they expressed, and use this as an indicator to improve meeting performance.

[1202] Below are some examples of prompts to input to a generative AI model:

[1203] 1. When silence occurs during a meeting, notifications are sent to participants who are not concentrating or showing negative emotions to encourage them to speak up.

[1204] 2. As the meeting progresses, a summary of what was discussed is sent every 15 minutes to all participants' devices.

[1205] 3. After the meeting, minutes are automatically generated based on the content of the comments and the analysis data, and each speaker is scored, taking into account the results of the emotion engine.

[1206] In this way, the system facilitates smooth meeting progress and supports effective discussions that take into account the emotional state of participants.

[1207] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1208] Step 1:

[1209] Starting a meeting and capturing data

[1210] When a user presses the conference start button on the terminal, the terminal starts capturing the conference video and audio streams. These streams are sent to the server as they are. The input is video and audio data from the camera and microphone connected to the terminal, and the output is sent to the server as stream data in real time.

[1211] Specifically, when a user presses a button to join a conference, the camera captures video and the microphone captures audio, and then these are sent to the server.

[1212] Step 2:

[1213] Video and audio stream analysis

[1214] The server analyzes the received video and audio streams using a generative AI model, converting speech into text using speech recognition technology (input: audio stream, output: text data), and analyzing facial expressions and movements using facial and movement recognition algorithms (input: video stream, output: facial and movement data).

[1215] Specifically, when a user says, "What do you think about this proposal?", the voice data is converted into text and saved as text data saying, "What do you think about this proposal?" At the same time, the user's facial expressions and movements are extracted from the video stream and converted into data.

[1216] Step 3:

[1217] Emotional state analysis using emotion engine

[1218] The server uses an emotion engine to determine the participant's emotional state from the analyzed facial expressions and tone of voice. The emotion engine categorizes the emotion from the analysis results (input: facial expression and tone of voice data, output: emotional state data) and uses that data for the next step.

[1219] Specifically, the emotion engine analyzes the facial expressions and tone of voice of the user when they speak, and converts emotions such as "joy," "anger," and "surprise" into data.

[1220] Step 4:

[1221] Silence detection and prompting

[1222] The server monitors the audio data during the conference and detects silence for a certain period of time (for example, 10 seconds) (input: audio data, output: silence detection status). When silence is detected, participants with negative emotional states or participants with declining concentration are identified based on the analysis results of the emotion engine, and a notification is sent to those participants encouraging them to speak (input: emotional state data, output: notification encouraging them to speak).

[1223] Specifically, the server detects 10 seconds of silence and sends a notification to participants whose emotion engine indicates a negative emotional state, saying, "Please give your opinion next."

[1224] Step 5:

[1225] Meeting summary and distribution

[1226] The server summarizes the discussion every 15 minutes. Using a generative AI model, it extracts key topics and decisions from each comment and creates a summary (input: comment data, output: summary data). This is then sent to all participants' devices.

[1227] Specifically, the server summarizes the contents of the 15-minute meeting into the topics "proposing a new project" and "confirming the budget," and distributes the summary information to the terminals of all participants.

[1228] Step 6:

[1229] Automatic generation and scoring of meeting minutes after the meeting

[1230] After the meeting ends, the server automatically generates minutes based on the accumulated remarks and analysis data. Using a generative AI model, the server records the speaker's name, remarks, timing of remarks, and emotional state in detail (input: remarks and analysis data, output: minutes data). Furthermore, it scores each speaker based on the number and content of their remarks (input: remarks data, output: scoring data). This is then sent to each participant's device.

[1231] Specifically, the server creates detailed minutes such as "Speaker A: Regarding the proposal for a new project" and "Speaker B: Comments on the proposal," and then notifies each participant of the scoring results, such as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments."

[1232] (Application example 2)

[1233] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] The issues of declining work efficiency and ensuring worker safety in factories are extremely important. Conventional systems have difficulty monitoring and analyzing the progress of work, the concentration level of workers, and their emotional state in real time, and issuing appropriate instructions or warnings as needed. Another issue is that while efforts to improve work efficiency have been made, there has been a lack of efforts to promote breaks that take into account workers' fatigue and emotional state.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1236] In this invention, the server includes: means for receiving video and audio streams during a conference; means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; means for sending notifications to encourage participants to speak; means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion; means for sending the summarized conference summaries to all participants' terminals; means for automatically generating minutes after the conference based on all comments and analysis data; means for scoring each participant based on the number of comments and the content of their comments; means for individually sending the scored results based on the number of comments and the content of their comments; means for monitoring work in real time and issuing a warning when a specific movement is detected; means for analyzing the concentration and emotional state of workers and encouraging them to take a break as necessary; and means for instructing the next work step when efficiency drops, thereby improving work efficiency in factories and ensuring worker safety.

[1237] A "video stream" refers to video data that is continuously sent from a device such as a camera.

[1238] An "audio stream" refers to a continuous stream of audio data sent from a microphone or other audio input device.

[1239] The term "analysis means" refers to a method or device for analyzing the content of speech, facial expressions, and actions of participants based on the video stream and audio stream.

[1240] "Means for detecting silence" refers to a method or device for detecting a period of no speech.

[1241] "Notification means" refers to a method or device for notifying participants of analysis results, warnings, instructions, etc.

[1242] "Means for summarizing the meeting" refers to a method or device for periodically summarizing what has been said and the progress of discussions during a meeting.

[1243] "Means for automatically generating minutes" refers to a method or device for creating minutes after a meeting based on the content of remarks and analysis data.

[1244] "Scoring means" refers to a method or device for evaluating each participant based on the number of comments and the content of their comments.

[1245] "Real-time monitoring means" refers to a method or device for monitoring and analyzing the real-time progress of work.

[1246] "Alert means" refers to a method or device for issuing an alert when a particular action or condition is detected.

[1247] "Means to encourage breaks" refers to methods or devices for analyzing a worker's concentration level and emotional state and instructing them to take a break as necessary.

[1248] "Means for indicating work steps" refers to a method or device for indicating the next work procedure when work efficiency drops.

[1249] The present invention relates to a work supervision system that aims to improve work efficiency in a factory and ensure the safety of workers. A specific embodiment of this system will be described below.

[1250] System configuration

[1251] Hardware configuration:

[1252] Smart glasses: Devices worn by workers that capture visual and audio information.

[1253] Server: A high-performance computer that analyzes collected data and generates instructions and warnings.

[1254] Camera: A device installed in each work area in a factory that provides a video stream.

[1255] Microphone: A device used to capture the voice of workers.

[1256] Software configuration:

[1257] Generative AI model (e.g., GPT-3): Converts input voice data into text and analyzes what is being said.

[1258] Emotion analysis engine (e.g., Emotion API): Analyzes facial expressions and movements from input video data to determine emotional state.

[1259] Streaming processing software (e.g., WebRTC): Sends and receives video and audio streams in real time.

[1260] Program processing

[1261] The server receives video and audio streams from the smart glasses and camera. The received data is analyzed in real time to analyze the worker's speech, facial expressions, and movements. This analysis is performed using a generative AI model and an emotion analysis engine. Based on the analysis results, the following processes are performed:

[1262] 1. Real-time monitoring: There is a way to monitor the progress of work and issue alerts when certain conditions are met (e.g., not wearing proper protective equipment).

[1263] 2. Promoting breaks: Analyze workers' concentration levels and emotional states and encourage them to take breaks as needed.

[1264] 3. Efficiency: There is a way to indicate the next work step if work efficiency declines.

[1265] Specific examples

[1266] Example 1: When worker A's concentration level is low

[1267] Based on the results of the server's analysis, the smart glasses will notify the user, "Your concentration level is declining. Please take a break."

[1268] Example 2: When worker B is not wearing protective equipment

[1269] The smart glasses notify workers to wear protective equipment, encouraging them to work safely.

[1270] Prompt Sentence Examples

[1271] Example prompt for generative AI model:

[1272] "Design an AI system to manage the work of the next shift. To ensure worker safety and efficiency, the system should include the following elements: facial expression analysis, voice recognition, emotion analysis, and real-time warning instructions. The following are specific prompts for analysis. The proposed configuration includes: when a worker is not wearing protective gear, when work efficiency drops sharply, how to issue appropriate instructions, how to communicate data between the server and the smart glasses, and what to display on the smart glasses."

[1273] The above is a specific embodiment of the present invention. This system realizes improved efficiency and safety in factory work.

[1274] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1275] Step 1:

[1276] The smart glasses and camera capture video and audio streams and send them to a server.

[1277] Input: Video and audio streams from smart glasses and camera.

[1278] Output: Sending video and audio streams to the server.

[1279] Step 2:

[1280] The server analyzes the received video and audio streams using a generative AI model and emotion analysis engine.

[1281] Input: Video and audio streams.

[1282] Output: Analysis data on each participant's speech, facial expressions, and movements.

[1283] Specific operations: Converting voice data into text and analyzing spoken content using a generative AI model, and analyzing facial expressions and movements using an emotion analysis engine.

[1284] Step 3:

[1285] The server monitors the progress of work based on the analysis results and detects specific actions or conditions.

[1286] Input: Analysis data.

[1287] Output: The result of detecting a specific behavior or state.

[1288] Specific actions: Checking that workers are wearing the correct protective equipment, detecting facial expressions such as fatigue or impatience.

[1289] Step 4:

[1290] The server sends an alert notification to the smart glasses when certain conditions are met.

[1291] Input: The detection of a specific action or state.

[1292] Output: Alert notification to smart glasses.

[1293] Specific operation: If a worker is not wearing protective equipment, the system notifies the worker to "put on protective equipment."

[1294] Step 5:

[1295] The server analyzes the worker's concentration level and emotional state and encourages them to take a break if necessary.

[1296] Input: Analysis data.

[1297] Output: Break promotion notification to smart glasses.

[1298] Specific behavior: If a worker's concentration level is decreasing, the system will notify them, "Your concentration level is decreasing. Please take a break."

[1299] Step 6:

[1300] The server instructs the next work step when the work efficiency drops.

[1301] Input: Data about work progress and efficiency.

[1302] Output: Next work step instructions to the smart glasses.

[1303] Specific behavior: If work is stalled, notify the user to "proceed to the next work step."

[1304] Through the above steps, efficiency and safety of work in the factory are improved.

[1305] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1306] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1307] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1308] [Fourth embodiment]

[1309] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1310] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1311] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1312] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1313] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1314] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1315] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1316] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1317] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1318] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1319] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1320] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1321] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1322] The system of the present invention is designed to analyze the video and audio streams of a conference in real time and facilitate the progress of the conference. A specific embodiment of this system will be described below.

[1323] The server receives the video and audio streams of the conference. It then analyzes the received video and audio streams to analyze the participants' speech, facial expressions, and movements in real time. Based on the analysis results, the server detects silence and identifies participants who should be encouraged to speak. It also sends notifications to encourage them to speak.

[1324] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The summarized meeting summary is sent to the devices of all participants. After the meeting ends, minutes are automatically generated based on all comments and analysis data. Each participant is scored based on the number of comments and the content of their comments, and the results are sent individually.

[1325] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[1326] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[1327] The operation of the system will be explained below using specific examples.

[1328] Example 1: Silence detection and prompting

[1329] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be prompted to speak next. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who then gives their opinion based on it.

[1330] Example 2: Meeting summary and sharing

[1331] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[1332] Example 3: Minutes and scoring

[1333] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[1334] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[1335] The processing flow will be explained below.

[1336] Step 1:

[1337] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[1338] Step 2:

[1339] The server analyzes the received video and audio streams. Specifically, it uses a generative AI model to analyze participants' speech, facial expressions, and movements in real time. Speech is converted into text, and facial expressions and movements are digitized as analysis results.

[1340] Step 3:

[1341] The server detects silence during the conference based on the analysis results. If no speech is made for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings.

[1342] Step 4:

[1343] When the server detects silence, it prompts the most appropriate participant to speak. It identifies participants who are not concentrating or appear sleepy from the analysis results and sends them a notification to prompt them to speak.

[1344] Step 5:

[1345] The device displays the notification received from the server to the user. Specifically, it displays a message such as "Please state your opinion next" on the screen to encourage the user to speak.

[1346] Step 6:

[1347] The user checks the notification displayed on the device and follows the instructions to express their opinions or ask questions. The comments are then sent back to the server via the device.

[1348] Step 7:

[1349] The server summarizes the meeting at regular intervals (for example, 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. This summary is generated in text format.

[1350] Step 8:

[1351] The server sends a summary of the meeting to all participants' terminals, which receive the summary message and display it to the user.

[1352] Step 9:

[1353] As the meeting progresses, the server continuously records what is said and the analytical data, so that all comments and discussions are stored in a log.

[1354] Step 10:

[1355] After the meeting, the server automatically generates minutes based on the accumulated remarks and analysis data. Specifically, it organizes the remarks made by each speaker and important points of discussion.

[1356] Step 11:

[1357] The server scores each participant based on the number and content of their comments. For example, the evaluation score is calculated based on multiple indicators such as the number of comments, the quality of the comments, and the degree of contribution to the discussion.

[1358] Step 12:

[1359] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[1360] Step 13:

[1361] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement.

[1362] The above are the specific processing steps of the present system based on the claims. This system allows meetings to proceed smoothly and enables highly productive discussions.

[1363] Example 1

[1364] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] Conventional conference systems lack sufficient support functions to facilitate smooth meeting progress, making it difficult for all participants to participate effectively in discussions. Furthermore, recording comments, generating minutes, and analyzing the content of comments must be done manually, which requires a great deal of time and effort. Another issue is that there are few ways to grasp participants' own contributions to the discussion, making it difficult to obtain specific feedback to improve the quality and efficiency of the meeting.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1367] In this invention, the server includes means for receiving video data and audio data during a conference, means for analyzing the received video data and audio data and analyzing the participants' comments, facial expressions, and movements in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion, means for transmitting the summarized conference summaries to all participants' terminals, means for automatically generating minutes after the conference based on all comments and the analysis data, means for scoring each participant based on the number of comments and the content of their comments, means for individually transmitting the scoring results based on the number of comments and the content of their comments, means for using computer vision technology to analyze the video data, means for using voice recognition technology to analyze the audio data, means for monitoring the audio level to detect silence for a certain period of time, means for notifying all participants of the generated conference summary, and means for distributing the minutes and scoring results to all participants.This allows the conference to proceed smoothly and allows all participants to participate efficiently in the discussion. In addition, automated recording and analysis of meetings reduces effort and time, and allows each participant to understand their own contribution and receive specific feedback that can help improve their meeting skills.

[1368] "Video data during a meeting" refers to visual information acquired through a camera while the meeting is in progress, and includes visual content such as the movements and facial expressions of participants.

[1369] "Audio data" refers to acoustic information acquired through a microphone during a conference, including speeches made by participants and other audio information.

[1370] "Means for receiving" refers to devices or software that provide the function of acquiring video data and audio data and inputting them into the server.

[1371] "Means for analyzing" refers to an algorithm or computer program that processes the received data and extracts specific information.

[1372] "Real-time analysis" refers to the act of processing acquired data immediately to obtain results, providing information immediately without delay.

[1373] "Silence detection means" means a device or software that provides the functionality to identify and detect periods of silence within audio data.

[1374] "Means for identifying participants who should be encouraged to speak" refers to algorithms or computer programs that use the analyzed data to identify the next person to speak.

[1375] "Means for sending notifications" refers to machines and programs for sending messages or alerts to specific participants.

[1376] A "meeting summary tool" refers to an algorithm or computer program that concisely summarizes what was discussed during a meeting.

[1377] "Means for sending a conference summary to all participants' terminals" refers to machines and programs that distribute messages to convey summarized information to all participants.

[1378] "Means for automatically generating minutes" refers to algorithms or computer programs that automatically create detailed records of meetings based on what was said during the meeting and analytical data.

[1379] "Means for scoring" refers to an algorithm or computer program that evaluates the number and content of each participant's comments and calculates a score based on that.

[1380] "Means for individual transmission" refers to a machine and program that provides the functionality for transmitting scoring results to each participant's terminal in a specific format.

[1381] "Computer vision technology" refers to a collection of techniques and algorithms used to extract information from digital images and video.

[1382] "Speech recognition technology" refers to a collection of techniques and algorithms for analyzing voice data and converting it into text.

[1383] "Means for monitoring audio levels" refers to an algorithm or computer program that continuously monitors the volume of audio data and detects fluctuations in the volume.

[1384] The system of the present invention is designed to analyze the video and audio streams of a conference in real time to facilitate the progress of the conference. A specific embodiment of this system will be described below, with each role played by a server, a terminal, and a user.

[1385] Server Roles

[1386] The server processes data and provides functions to support the conference through the following procedures:

[1387] 1. Receiving data:

[1388] The server receives video and audio data during the conference using a communication protocol such as WebRTC, which transmits real-time video and audio streams from each participant to the server.

[1389] 2. Video and audio data analysis:

[1390] The received video data is analyzed using OpenCV and other computer vision technologies, specifically using facial recognition and expression analysis algorithms to understand the participants' facial expressions and movements.

[1391] The voice data is converted into text using voice recognition technology (for example, Google Cloud Speech-to-Text API), which makes the spoken content available as text data.

[1392] 3. Silence detection and prompting:

[1393] The server continuously monitors the audio level and detects silence for a certain period (for example, 10 seconds) using a specific algorithm that monitors the audio level.

[1394] When silence is detected, an algorithm determines who should be the next person to speak and sends them a notification, such as "You're next."

[1395] 4. Summary of the meeting:

[1396] Based on what was said during the meeting and the progress of the discussion, a generative AI model is used to summarize the main topics and decisions of the meeting at regular intervals (for example, every 15 minutes).

[1397] A summary of the meeting is sent to all participants' devices, making it easier for them to understand the flow of the meeting.

[1398] 5. Generate transcripts:

[1399] After the meeting, minutes are automatically generated based on all statements and analysis data. The generated minutes include a detailed record of what each speaker said.

[1400] 6. Scoring:

[1401] Each participant is scored based on the number of comments and the content of their comments. This scoring is based on the evaluation criteria of the frequency, importance, and contribution of comments.

[1402] The scored results are sent individually to each participant's device.

[1403] Device Role

[1404] The terminal provides the user with visual and auditory information about the data sent from the server, and transmits user operations to the server.

[1405] 1. Displaying data:

[1406] The terminal displays the video and audio streams sent from the server, allowing users to view the contents of the conference in real time.

[1407] 2. Capturing what users say and do:

[1408] When a user speaks, the device captures the user's voice and sends it to the server. It also captures the user's facial expressions and movements and sends this data to the server for analysis.

[1409] 3. Notification and Summary Display:

[1410] When a notification encouraging speaking or a summary of the conference is sent from the server, the terminal displays this to the user as a pop-up notification or message.

[1411] User Roles

[1412] Users can participate in the conference and make comments and hold discussions as usual. In addition, they can check notifications and summaries from the server and use them to express their opinions and advance the discussion.

[1413] 1. Statement:

[1414] The user speaks through a microphone, and the content of the speech is transmitted to the server via the terminal.

[1415] Facial expressions and movements are also captured by the camera and sent to the server.

[1416] 2. Responding to Notices:

[1417] When a user receives a notification from the server prompting the user to speak, the user responds by speaking.

[1418] 3. Review of minutes and scoring results:

[1419] After the meeting, users can check the minutes and scoring results displayed on their device to help improve their own meeting skills.

[1420] Specific examples

[1421] Example 1: Silence detection and prompting

[1422] During a meeting, if a participant finishes speaking and there is a 10-second silence, the server detects the silence and identifies the next participant to be prompted to speak. For example, if the analysis results indicate that a particular participant is sleepy, the server sends that participant a notification saying, "Please give your opinion next." The device displays this notification to the user, who can then express their opinion.

[1423] Example 2: Meeting Summary

[1424] As the meeting progresses, the server summarizes the discussions every 15 minutes. For example, it summarizes the main topics and decisions made during the first 15 minutes and sends the summary to all participants' devices. The devices then display this summary information to the users, allowing them to keep track of the progress of the meeting.

[1425] Example 3: Minutes and scoring

[1426] After the meeting ends, the server automatically generates minutes based on all comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding the new project proposal" and "Speaker B: Comments on the proposal." Furthermore, each speaker is scored based on the number of comments and the content of their comments. For example, an evaluation might be made as "Speaker A: 10 comments, 3 important comments" or "Speaker B: 5 comments, 2 supplementary comments." These results are sent individually to each participant's device, allowing users to check their own performance.

[1427] The above is a specific embodiment of the present invention. This system will facilitate smooth progress in meetings, allowing all participants to participate in discussions efficiently. It will also improve the quality of meetings and help improve individual meeting skills.

[1428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1429] Step 1:

[1430] Receiving data

[1431] The server receives video and audio data during the conference using the WebRTC protocol.

[1432] Input: Real-time video and audio streams sent from each participant's device.

[1433] Data processing: Video and audio data are received in the form of packets.

[1434] Output: Received video and audio data.

[1435] How it works: The server uses the WebRTC protocol to receive video and audio streams from each participant in real time, store them in a database, and queue them for processing.

[1436] Step 2:

[1437] Video data analysis

[1438] The server analyzes the received video data using the OpenCV library.

[1439] Input: Video data received via the WebRTC protocol.

[1440] Data processing: Applying facial recognition and expression analysis algorithms to extract specific facial features and changes in facial expressions.

[1441] Output: Participant's facial expression and movement data.

[1442] Specific operation: The server uses OpenCV to detect facial features for each video frame and analyzes changes in facial expression and eye movements, thereby classifying facial expressions into categories such as "smiling," "serious," and "sleepy."

[1443] Step 3:

[1444] Analysis of audio data

[1445] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[1446] Input: Audio data received via the WebRTC protocol.

[1447] Data processing: Using voice recognition technology, voice data is converted into text data.

[1448] Output: Text data of the speech.

[1449] What it does: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts what is said into text in real time, which is then stored in a database for further analysis and recording.

[1450] Step 4:

[1451] Silence Detection

[1452] The server monitors the analyzed audio data and detects periods of silence.

[1453] Input: Speech and speech level data converted to text.

[1454] Data processing: Based on the voice level data, monitor whether there is any continuous speech for a specific period (for example, 10 seconds).

[1455] Output: Silence detection results.

[1456] What happens: The server checks that the audio level is below a certain threshold and then runs an algorithm that detects silence if no speech is heard for 10 consecutive seconds.

[1457] Step 5:

[1458] Sending a call to speak

[1459] If a period of silence continues, the server identifies the participant who should next be prompted to speak and sends a notification.

[1460] Input: Silence detection results and analyzed facial expression data.

[1461] Data processing: Based on the analysis results obtained from video and audio data, participants who should be encouraged to speak are identified.

[1462] Output: Informational message.

[1463] Specific operation: The server sends a notification to the identified participant's device saying, "Please give your opinion next." This notification is displayed as a pop-up message on the device.

[1464] Step 6:

[1465] Conference Summary

[1466] The server periodically summarizes the contents of the conference.

[1467] Input: Text data converted from audio data and discussion progress.

[1468] Data processing: Using NLP techniques, we analyze what is being said and extract key topics and decisions.

[1469] Output: Meeting summary text.

[1470] Specific operation: The server uses a generative AI model to summarize the contents of the meeting every 15 minutes and sends the summary text to the devices of all participants.

[1471] Step 7:

[1472] Generate meeting minutes

[1473] After the meeting ends, the server automatically generates minutes based on all statements and analysis data.

[1474] Input: Text data and analysis data of all statements.

[1475] Data processing: Integrate the statements and analytical data and organize them into detailed records.

[1476] Output: Auto-generated meeting transcript.

[1477] Specific operation: The server automatically generates minutes based on all data collected during the meeting and sends them to each participant's device. The minutes include the remarks and decisions made by each participant.

[1478] Step 8:

[1479] Sending scoring results

[1480] The server scores each participant based on the number of comments and the content of their comments, and sends the results individually.

[1481] Input: All speech content and number of speeches data.

[1482] Data processing: A score is calculated for each speaker based on the number of comments, importance, and contribution of each speaker.

[1483] Output: Scoring results.

[1484] How it works: The server evaluates and scores each participant's performance, taking into account the frequency of their comments and the importance of their content. The results are sent to each participant's device as individual notifications.

[1485] These are the specific processing steps of the system. This allows meetings to proceed smoothly and enables all participants to participate in the discussion efficiently. In addition, the automatic generation of meeting minutes and feedback of scoring results can contribute to improving the quality of meetings and the skills of participants.

[1486] (Application example 1)

[1487] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1488] Conventional conference systems have difficulty efficiently and effectively managing tasks such as encouraging participants to speak and creating meeting minutes. In particular, current conference management systems are insufficient in production sites such as factories, where rapid decision-making and communication are required. Furthermore, there is no system for analyzing participants' facial expressions and movements in real time, making it difficult to properly grasp the status of participants during the meeting, which can slow down the progress of the meeting. Therefore, an effective system to solve these issues is needed.

[1489] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1490] In this invention, the server includes means for receiving video and audio streams during a conference, means for analyzing the received video and audio streams and analyzing the speech content, facial expressions, and movements of participants in real time, means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak, means for sending notifications to encourage them to speak, means for analyzing the video and audio streams and encouraging specific participants to speak to support in-factory conferences, and means for applying a facial expression analysis model using the video and audio streams, identifying facial expressions based on the analysis results, and determining participants who should be encouraged to speak, thereby enabling quick decision-making and efficient communication within the factory.

[1491] A "video stream" is a series of video data captured during a conference via a video input device such as a camera.

[1492] An "audio stream" is a series of audio data captured during a conference via an audio input device such as a microphone.

[1493] "Real-time analysis" refers to the process of analyzing captured video and audio streams in real time and providing results without delay.

[1494] "Silence detection" refers to determining whether a silence state has occurred for a certain period of time or longer based on the analyzed audio stream.

[1495] "Speaker prompt notification" is a function that sends a message to specific participants to prompt them to speak when silence is detected.

[1496] The "facial expression analysis model" is a machine learning model for analyzing participants' facial expressions obtained from video streams and estimating their emotional states.

[1497] The "analysis results" are information on speech content, facial expressions, and movements obtained from the video stream and audio stream data.

[1498] "Scoring" is the process of evaluating each participant based on the number and content of their comments.

[1499] Minutes are documents that record what was said and what was decided at a meeting.

[1500] "Factory meetings" are meetings held in factories for the purpose of planning and solving problems related to production.

[1501] "Notification means" is a function for sending messages and instructions from the system to participants.

[1502] "Multimodal analysis" is the process of simultaneously analyzing multiple different data sources (such as video and audio streams) and utilizing the results in a comprehensive manner.

[1503] The present invention provides a system for smoothly conducting meetings in a factory and promoting efficient decision-making. Specific embodiments for implementing this system will be described below.

[1504] System Overview

[1505] This system consists of three roles: a server, a terminal, and a user. The server plays a central role in receiving and analyzing the video and audio streams of the conference. The terminal is a device through which participants interact with the system, displaying the video and audio streams and receiving notifications. The user participates in the conference, expressing their opinions and participating in discussions.

[1506] Hardware and Software Configuration

[1507] Hardware: Camera, microphone, display, smartphone.

[1508] Software: Python, OpenCV (video stream processing), TensorFlow (facial expression analysis model), Numpy (data processing).

[1509] Data processing and calculation

[1510] Server Processing

[1511] 1. Receiving video and audio streams

[1512] The server receives the video and audio streams from the camera and microphone in real time.

[1513] 2. Real-time analysis

[1514] We use OpenCV to detect participants' faces from the video stream, TensorFlow to analyze detected facial expressions and estimate participants' emotional states, and Numpy to analyze the volume levels of the audio stream and detect silence.

[1515] 3. Detect silence and send notifications

[1516] If silence continues for a certain period of time, the system will identify the participant who should be prompted to speak next based on the analysis results and send a notification to that participant, which will be displayed on their device.

[1517] 4. Meeting summary and minutes generation

[1518] Based on the main comments made during the meeting and the progress of the discussion, a summary of the meeting is generated at regular intervals (for example, every 15 minutes) and sent to the devices of all participants. After the meeting, minutes are automatically generated based on all comments and analysis data, and the number and content of comments made by each participant are scored.

[1519] Terminal handling

[1520] 1. Displaying video and audio streams

[1521] The terminal displays the video and audio streams sent from the server, allowing participants to visually understand the contents of the conference.

[1522] 2. Receiving and Displaying Notifications

[1523] It receives notifications from the server and displays them to users, who can then express their opinions and advance the discussion.

[1524] Specific examples

[1525] If there is silence during a meeting, the system will:

[1526] 1. At the beginning of the meeting, the statement is made, "We will begin discussing the design of the new product."

[1527] 2. If there is a period of silence, the server detects this and identifies the next participant to speak.

[1528] 3. A notification will appear on the device of the person you are speaking to saying, "Yamada-san, please comment on the next steps."

[1529] 4. This allows Yamada to express his opinion and the meeting proceeds smoothly.

[1530] An example prompt is:

[1531] Meeting start: "We're going to start discussing the design of our new product."

[1532] There is silence for a while.

[1533] Prompt to speak: "Yamada-san, please comment on next steps."

[1534] In this way, this system enables smooth progress in factory meetings and efficient decision-making.

[1535] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1536] Step 1:

[1537] Receiving video and audio streams

[1538] The server receives video and audio streams in real time through cameras and microphones. The input is the video and audio data of the meeting, and the output is the raw data stored and prepared for analysis. In this step, the server continuously acquires data by capturing the streams from each device.

[1539] Step 2:

[1540] Real-time analysis of video streams

[1541] The server uses OpenCV to detect participants' faces from the video stream. The input is the received video stream data, and the output is the location information of the detected faces. This face location information becomes the input for subsequent facial expression analysis. Specifically, the server recognizes the contours of the face from the video frame and identifies its location.

[1542] Step 3:

[1543] Application of facial expression analysis model

[1544] The server performs facial expression analysis on the detected faces using TensorFlow. The input is image data of salient faces and a pre-trained facial expression analysis model, and the output is an estimated emotional state (e.g., joy, anger, sadness, happiness, etc.). This is the process of applying the facial expression analysis model to classify facial expressions.

[1545] Step 4:

[1546] Parsing an Audio Stream

[1547] The server uses Numpy to analyze the volume level of the audio stream and detect silence. The input is the audio stream data acquired in real time, and the output is the volume level (silence or not) judgment result. Specifically, it calculates the amplitude of the audio data and determines silence if it is below a preset threshold.

[1548] Step 5:

[1549] Detect silence and send prompts

[1550] If silence continues for a certain period of time (e.g., 10 seconds), the server identifies the participant who should be prompted to speak next based on the results of facial expression analysis. The input is the silence detection result and the results of facial expression analysis, and the output is the identified participant ID and the corresponding prompt to speak. The server then sends a "please speak next" notification to this identified participant.

[1551] Step 6:

[1552] Viewing video and audio streams

[1553] The terminal receives the video and audio streams sent from the server and displays them to the user. The input is the video and audio stream data from the server, and the output is the display of that data on the terminal. This allows the user to grasp the status of the conference in real time.

[1554] Step 7:

[1555] Receiving and Viewing Notifications

[1556] The terminal receives a prompt to speak from the server and displays it to the user. The input is the prompt to speak notification data from the server, and the output is the display of the notification to the user. Upon receiving this notification, the user can express their opinion or advance the discussion.

[1557] Step 8:

[1558] Generate meeting summaries and minutes

[1559] The server generates a summary of the meeting at regular intervals (e.g., every 15 minutes) based on the main comments made during the meeting and the progress of the discussion, and sends it to the devices of all participants. After the meeting ends, it automatically generates minutes based on all comments and analysis data, scoring the number and content of comments made by each participant. The input is all comment data and analysis results from the meeting, and the output is summary information and minutes. Specifically, it uses natural language processing technology to create summaries, extract necessary information, and document it.

[1560] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1561] The present invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. Specific embodiments of this system are described below.

[1562] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows participants' speech, facial expressions, actions, and emotions to be analyzed in real time. Speech is converted into text, and facial expressions and actions are converted into data as analysis results. The emotion engine also recognizes participants' emotional states from their facial expressions and tone of voice.

[1563] Based on the analysis results, the server detects silence during the meeting. If no one speaks for a certain period of time (for example, 10 seconds), the server determines this as silence. This period can be changed in the system settings. When silence is detected, the server also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. For example, it sends a notification encouraging participants who are determined to be not concentrating or whose emotional state is changing negatively to speak.

[1564] The terminal displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the content is captured and sent to the server. The terminal also captures the user's facial expressions and movements and sends them to the server. Furthermore, if a notification encouraging the user to speak or a summary of the conference is sent, the terminal displays it to the user.

[1565] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[1566] The operation of the system will be explained below using specific examples.

[1567] Example 1: Silence detection and prompting

[1568] During a meeting, suppose a participant finishes speaking and then there is 10 seconds of silence. In this case, the server detects the silence and identifies the participant who should be encouraged to speak next. If the emotion engine's analysis determines that a particular participant is losing concentration or showing negative emotions, it sends that participant a notification saying, "Please speak next." The device displays this notification to the user, who then expresses their opinion based on it.

[1569] Example 2: Meeting summary and sharing

[1570] As the meeting progresses, the server summarizes the content of the discussion every 15 minutes. For example, it briefly summarizes the main topics and decisions discussed in the first 15 minutes and sends this content to the devices of all participants. The analysis results of the emotion engine are also taken into account, and summary information corresponding to increases in negative emotions and decreases in positive emotions is provided. The device displays this summary information to the user, allowing them to always understand the progress of the meeting.

[1571] Example 3: Minutes and scoring

[1572] After the meeting ends, the server automatically generates minutes based on the accumulated content of comments and analysis data. For example, a detailed record is created in the form of "Speaker A: Regarding a new project proposal" and "Speaker B: Opinions on the proposal." The analysis results of the emotion engine are also reflected, and the emotional state of each speaker is included in the minutes. In addition, each speaker is scored based on the number of comments and content of their comments. For example, they might be evaluated as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments." The scoring also takes into account the quality of the comments and the emotional state. These results are sent individually to each participant's device, allowing users to check their own performance.

[1573] The above is a specific embodiment of the present invention. This system not only allows meetings to proceed smoothly, but also allows for more effective discussions by taking into account the emotional states of participants. The introduction of an emotion engine will improve the quality of meetings and also help improve individual meeting skills.

[1574] The processing flow will be explained below.

[1575] Step 1:

[1576] The server receives the video and audio streams of the conference, with the video and audio data of the participants being transferred to the server in real time.

[1577] Step 2:

[1578] The server uses generative AI models and emotion engines to analyze the incoming video and audio streams in real time, converting participants' speech into text and digitizing their facial expressions, movements, and emotional states.

[1579] Step 3:

[1580] Based on the analysis results, the server monitors speech content, facial expressions, and movements to detect silence. If no speech is heard for a certain period of time (for example, 10 seconds), the server determines this as silence.

[1581] Step 4:

[1582] When the server detects silence, it also takes into account the analysis results of the emotion engine and encourages the most appropriate participant to speak. If a specific participant is losing concentration or showing negative emotions, it sends a notification encouraging them to speak. At this time, an appropriate message is selected depending on the situation.

[1583] Step 5:

[1584] The terminal displays the notification received from the server to the user. Specifically, a message such as "Please give your opinion next" is displayed on the screen of the user terminal.

[1585] Step 6:

[1586] Users can check the notifications displayed on their devices and express their opinions or questions. The content of their comments is captured by the device and sent to the server in real time.

[1587] Step 7:

[1588] The server summarizes the meeting at regular intervals (for example, every 15 minutes) based on the content of comments made during the meeting and the progress of the discussion. The server also reflects the analysis results of the emotion engine, and generates summary information that takes into account changes in emotions.

[1589] Step 8:

[1590] The server sends a summary of the conference to all participants' terminals, which receive the summary information and display it to the user, allowing the user to keep track of the conference progress.

[1591] Step 9:

[1592] The server records the content of comments and analysis data that is accumulated in real time, and stores the data necessary for creating minutes after the meeting.

[1593] Step 10:

[1594] After the meeting, the server automatically generates minutes based on the saved remarks and analysis data, including the remarks made by each speaker, key discussion points, and emotional state.

[1595] Step 11:

[1596] The server scores each participant based on the number and content of their comments, which reflects the quality, quantity, and emotional state of their comments.

[1597] Step 12:

[1598] The server sends the generated minutes and scoring results to each participant's device, which receives this information and displays it to the user.

[1599] Step 13:

[1600] Users can check the minutes and scoring results on their devices, which allows them to evaluate their own comments and contributions to the meeting and identify areas for improvement in the future.

[1601] These are the specific processing steps of this system, which combines an emotion engine. This system ensures smooth progress in meetings and highly productive discussions. Furthermore, by taking into account the emotional state of participants, the quality of meetings can be further improved.

[1602] Example 2

[1603] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1604] Conventional conferencing systems lack the means to effectively analyze participants' remarks, facial expressions, actions, and emotions in real time, resulting in issues such as the inability to smoothly progress through the meeting and the inability to respond to changes in participants' concentration and emotional state. Furthermore, there are insufficient means to prompt appropriate participants to speak when silence persists, resulting in a decrease in meeting efficiency. Furthermore, summarizing discussions during meetings and creating meeting minutes after the meeting are often done manually, which is time-consuming and labor-intensive.

[1605] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1606] In this invention, the server includes: a means for receiving video and audio streams during a conference; a means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; an emotion engine including means for determining participants' emotional states from facial expressions and tone of voice; a means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; a means for sending notifications encouraging participants who show a negative emotional state or a lack of concentration to speak; a means for summarizing the conference at regular intervals based on the content of comments made during the conference and the progress of the discussion; a means for sending the summarized conference summary to all participants' terminals; a means for automatically generating minutes after the conference based on all comments and analysis data; a means for scoring each participant based on the number of comments and the content of their comments; and a means for individually transmitting the scored results. This not only facilitates smooth progress in the conference, but also takes into account the participants' emotional states, enabling more effective discussions. Furthermore, the creation of summaries during the conference and the automatic generation of minutes after the conference ends significantly reduces time and effort.

[1607] 1. "Video Stream" means a data stream for transmitting and receiving video of participants in real time during a conference.

[1608] 2. "Audio Stream" means a data stream for transmitting and receiving the voices of participants in a conference in real time.

[1609] 3. "Analysis" refers to the process of decomposing and processing the content of received video and audio streams to extract various information.

[1610] 4. "Speech content" refers to the text information spoken by each participant during the meeting.

[1611] 5. “Facial expression” refers to information about a participant’s emotional state that can be read from their facial movements and expressions.

[1612] 6. "Movement" refers to information about participants' body movements and gestures.

[1613] 7. "Emotional state" refers to the psychological state of the participant as analyzed based on facial expressions and tone of voice.

[1614] 8. The "Emotion Engine" is software that analyzes and determines participants' emotional state from facial expressions and tone of voice.

[1615] 9. "Silence detection" is the process of automatically detecting periods of inactivity during a meeting.

[1616] 10. A "prompt to speak" is a message sent to prompt a specific participant to speak next when there is a period of silence.

[1617] 11. A "meeting summary" is a concise summary of the major topics and decisions discussed during the meeting.

[1618] 12. "Summarizing" refers to extracting important points based on what was said during a meeting and the progress of the discussion, and summarizing them in a concise form.

[1619] 13. "Minutes" means a document that contains a detailed record of what was discussed, what decisions were made, and what each speaker said at a meeting.

[1620] 14. "Scoring" is the process of evaluating each participant based on the number of comments and the quality of their comments.

[1621] 15. "Multimodal analysis" refers to the technique of simultaneously analyzing multiple data formats (e.g., audio and video).

[1622] 16. "Timer means" means a device or software that measures a certain period of time and monitors whether or not any speech is made during that period.

[1623] This invention relates to a system that analyzes speech content, facial expressions, and actions during a meeting in real time, and further combines it with an emotion engine to facilitate the progress of the meeting. A specific embodiment of this system is described below.

[1624] The server receives the video and audio streams of the conference. It uses a generative AI model and an emotion engine to analyze the received video and audio streams. This allows for real-time analysis of participants' speech, facial expressions, actions, and emotions. Specifically, speech is converted into text using speech recognition technology. Facial and motion recognition algorithms are used to analyze facial expressions and actions. The emotion engine determines participants' emotional states from their facial expressions and tone of voice. Based on the results of this analysis, the server detects silence and prompts participants who show negative emotional states or low concentration to speak.

[1625] For example, if a 10-second silence occurs during a meeting, participants who are showing signs of low concentration or negative emotions can be notified by the emotion engine's analysis, with the message "Please speak next." The device that receives this notification will display it to the user, encouraging them to speak next.

[1626] The device displays the video and audio streams sent from the server, providing the user with a visual representation of the conference content. When the user speaks, the device captures the content and sends it to the server. It also captures the user's facial expressions and movements and sends them to the server. Furthermore, if the server sends a notification encouraging the user to speak or a summary of the conference, the device displays it to the user.

[1627] Users participate in the meeting and make comments and participate in the discussion as usual. They check notifications and summaries from the server, and use them to express their opinions and advance the discussion. After the meeting ends, users can check the minutes and scoring results displayed on their devices, which can be used to improve their own meeting skills.

[1628] For example, the server summarizes the meeting every 15 minutes based on what is said during the meeting and the progress of the discussion, and sends this summary to the devices of all participants. If the meeting progresses on two topics, "proposing a new project" and "checking the budget," the server notifies the user of the summary. This allows users to always understand the progress of the meeting.

[1629] Next, after the meeting ends, the server automatically generates minutes based on the text data of comments and facial expression and emotion analysis data. For example, detailed minutes such as "Speaker A: Regarding the new project proposal" and "Speaker B: Opinions on the proposal" are created. Furthermore, each speaker is scored based on the number and content of their comments, taking into account the quality of their comments and their emotional state. Users can check the number of comments they made and the number of important opinions they expressed, and use this as an indicator to improve meeting performance.

[1630] Below are some examples of prompts to input to a generative AI model:

[1631] 1. When silence occurs during a meeting, notifications are sent to participants who are not concentrating or showing negative emotions to encourage them to speak up.

[1632] 2. As the meeting progresses, a summary of what was discussed is sent every 15 minutes to all participants' devices.

[1633] 3. After the meeting, minutes are automatically generated based on the content of the comments and the analysis data, and each speaker is scored, taking into account the results of the emotion engine.

[1634] In this way, the system facilitates smooth meeting progress and supports effective discussions that take into account the emotional state of participants.

[1635] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1636] Step 1:

[1637] Starting a meeting and capturing data

[1638] When a user presses the conference start button on the terminal, the terminal starts capturing the conference video and audio streams. These streams are sent to the server as they are. The input is video and audio data from the camera and microphone connected to the terminal, and the output is sent to the server as stream data in real time.

[1639] Specifically, when a user presses a button to join a conference, the camera captures video and the microphone captures audio, and then these are sent to the server.

[1640] Step 2:

[1641] Video and audio stream analysis

[1642] The server analyzes the received video and audio streams using a generative AI model, converting speech into text using speech recognition technology (input: audio stream, output: text data), and analyzing facial expressions and movements using facial and movement recognition algorithms (input: video stream, output: facial and movement data).

[1643] Specifically, when a user says, "What do you think about this proposal?", the voice data is converted into text and saved as text data saying, "What do you think about this proposal?" At the same time, the user's facial expressions and movements are extracted from the video stream and converted into data.

[1644] Step 3:

[1645] Emotional state analysis using emotion engine

[1646] The server uses an emotion engine to determine the participant's emotional state from the analyzed facial expressions and tone of voice. The emotion engine categorizes the emotion from the analysis results (input: facial expression and tone of voice data, output: emotional state data) and uses that data for the next step.

[1647] Specifically, the emotion engine analyzes the facial expressions and tone of voice of the user when they speak, and converts emotions such as "joy," "anger," and "surprise" into data.

[1648] Step 4:

[1649] Silence detection and prompting

[1650] The server monitors the audio data during the conference and detects silence for a certain period of time (for example, 10 seconds) (input: audio data, output: silence detection status). When silence is detected, participants with negative emotional states or participants with declining concentration are identified based on the analysis results of the emotion engine, and a notification is sent to those participants encouraging them to speak (input: emotional state data, output: notification encouraging them to speak).

[1651] Specifically, the server detects 10 seconds of silence and sends a notification to participants whose emotion engine indicates a negative emotional state, saying, "Please give your opinion next."

[1652] Step 5:

[1653] Meeting summary and distribution

[1654] The server summarizes the discussion every 15 minutes. Using a generative AI model, it extracts key topics and decisions from each comment and creates a summary (input: comment data, output: summary data). This is then sent to all participants' devices.

[1655] Specifically, the server summarizes the contents of the 15-minute meeting into the topics "proposing a new project" and "confirming the budget," and distributes the summary information to the terminals of all participants.

[1656] Step 6:

[1657] Automatic generation and scoring of meeting minutes after the meeting

[1658] After the meeting ends, the server automatically generates minutes based on the accumulated remarks and analysis data. Using a generative AI model, the server records the speaker's name, remarks, timing of remarks, and emotional state in detail (input: remarks and analysis data, output: minutes data). Furthermore, it scores each speaker based on the number and content of their remarks (input: remarks data, output: scoring data). This is then sent to each participant's device.

[1659] Specifically, the server creates detailed minutes such as "Speaker A: Regarding the proposal for a new project" and "Speaker B: Comments on the proposal," and then notifies each participant of the scoring results, such as "Speaker A: 10 comments, 3 important comments" and "Speaker B: 5 comments, 2 supplementary comments."

[1660] (Application example 2)

[1661] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1662] The issues of declining work efficiency and ensuring worker safety in factories are extremely important. Conventional systems have difficulty monitoring and analyzing the progress of work, the concentration level of workers, and their emotional state in real time, and issuing appropriate instructions or warnings as needed. Another issue is that while efforts to improve work efficiency have been made, there has been a lack of efforts to promote breaks that take into account workers' fatigue and emotional state.

[1663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1664] In this invention, the server includes: means for receiving video and audio streams during a conference; means for analyzing the received video and audio streams and analyzing participants' comments, facial expressions, and movements in real time; means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; means for sending notifications to encourage participants to speak; means for summarizing the conference at regular intervals based on the comments made during the conference and the progress of the discussion; means for sending the summarized conference summaries to all participants' terminals; means for automatically generating minutes after the conference based on all comments and analysis data; means for scoring each participant based on the number of comments and the content of their comments; means for individually sending the scored results based on the number of comments and the content of their comments; means for monitoring work in real time and issuing a warning when a specific movement is detected; means for analyzing the concentration and emotional state of workers and encouraging them to take a break as necessary; and means for instructing the next work step when efficiency drops, thereby improving work efficiency in factories and ensuring worker safety.

[1665] A "video stream" refers to video data that is continuously sent from a device such as a camera.

[1666] An "audio stream" refers to a continuous stream of audio data sent from a microphone or other audio input device.

[1667] The term "analysis means" refers to a method or device for analyzing the content of speech, facial expressions, and actions of participants based on the video stream and audio stream.

[1668] "Means for detecting silence" refers to a method or device for detecting a period of no speech.

[1669] "Notification means" refers to a method or device for notifying participants of analysis results, warnings, instructions, etc.

[1670] "Means for summarizing the meeting" refers to a method or device for periodically summarizing what has been said and the progress of discussions during a meeting.

[1671] "Means for automatically generating minutes" refers to a method or device for creating minutes after a meeting based on the content of remarks and analysis data.

[1672] "Scoring means" refers to a method or device for evaluating each participant based on the number of comments and the content of their comments.

[1673] "Real-time monitoring means" refers to a method or device for monitoring and analyzing the real-time progress of work.

[1674] "Alert means" refers to a method or device for issuing an alert when a particular action or condition is detected.

[1675] "Means to encourage breaks" refers to methods or devices for analyzing a worker's concentration level and emotional state and instructing them to take a break as necessary.

[1676] "Means for indicating work steps" refers to a method or device for indicating the next work procedure when work efficiency drops.

[1677] The present invention relates to a work supervision system that aims to improve work efficiency in a factory and ensure the safety of workers. A specific embodiment of this system will be described below.

[1678] System configuration

[1679] Hardware configuration:

[1680] Smart glasses: Devices worn by workers that capture visual and audio information.

[1681] Server: A high-performance computer that analyzes collected data and generates instructions and warnings.

[1682] Camera: A device installed in each work area in a factory that provides a video stream.

[1683] Microphone: A device used to capture the voice of workers.

[1684] Software configuration:

[1685] Generative AI model (e.g., GPT-3): Converts input voice data into text and analyzes what is being said.

[1686] Emotion analysis engine (e.g., Emotion API): Analyzes facial expressions and movements from input video data to determine emotional state.

[1687] Streaming processing software (e.g., WebRTC): Sends and receives video and audio streams in real time.

[1688] Program processing

[1689] The server receives video and audio streams from the smart glasses and camera. The received data is analyzed in real time t...

Claims

1. means for receiving video and audio streams during the conference; A means for analyzing the received video and audio streams and analyzing the participants' speech, facial expressions, and movements in real time; A means for detecting silence based on the analysis results and identifying participants who should be encouraged to speak; a means for sending notifications to encourage people to speak; A means for summarizing the outline of a meeting at regular intervals based on the content of statements made during the meeting and the progress of the discussion; means for transmitting a summarized meeting summary to all participants' terminals; After the meeting, a means to automatically generate minutes based on all statements and analysis data, A means for scoring each participant based on the number of comments and the content of their comments; A means for individually transmitting the results scored based on the number of comments and the content of the comments; A system including:

2. 2. The system of claim 1, wherein the means for analyzing the video and audio streams during the conference includes means for performing multimodal analysis and simultaneously analyzing facial expressions and movements of participants.

3. 2. The system of claim 1, wherein the means for detecting silence includes timer means for monitoring whether or not speech has occurred for a certain period of time.

Images