System

Abstract

To provide a system.SOLUTION: A system includes: means for automatically facilitating a video conference by artificial intelligence; means for converting audio data of the video conference into text using voice recognition technology; means for analyzing facial expressions from video data on the video conference using an image recognition technology; means for analyzing remarks, facial expressions, and tone of voice of attendees at the video conference and digitizing their emotions; means for optimally facilitating the conference on the basis of emotional data digitized by the digitizing means; and means for providing feedback on the basis of the progress of the conference and the emotional data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's society, where video conferencing is on the rise due to the spread of remote work, there is a need to facilitate non-face-to-face communication and improve the quality of meetings. However, existing video conferencing tools have the challenge of making it difficult to accurately grasp the emotions of participants and provide appropriate facilitation. [Means for solving the problem]

[0005] This invention provides a system that uses AI to automatically facilitate video conferences. It analyzes the content of participants' remarks, facial expressions, and tone of voice, and quantifies their emotions to provide optimal conference facilitation. It also provides specialized facilitation, such as patent-specific and strategy-specific facilitation, reducing the burden on users and improving the quality and effectiveness of meetings. This provides an environment where smooth communication can be achieved even without face-to-face meetings. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One aspect of the present invention is a system that uses AI to automatically facilitate video conferences. Specifically, it is equipped with an AI engine that analyzes the content, facial expressions, and tone of voice of attendees. This AI engine uses voice recognition and image recognition technology to quantify attendees' emotions, and based on the results, optimally facilitates the conference.

[0029] "Example 2"

[0030] Furthermore, it also has the ability to provide specialized facilitation, such as patent-specific and strategy-specific facilitation. For example, in patent-specific facilitation, AI automatically asks questions and makes suggestions to promote patent-related discussions. This reduces the burden on users and improves the quality and effectiveness of meetings.

[0031] "Example 3"

[0032] It also has a function that provides an environment where smooth communication can be carried out even without face-to-face meetings. Specifically, the AI ​​analyzes the emotions of attendees and adjusts the progress of the meeting based on the results. For example, if it determines that attendees' emotions are leaning negatively, the AI ​​can automatically perform appropriate facilitation like a human would, such as pausing the meeting and suggesting that attendees take a break.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The AI ​​engine starts the video conference and analyzes the participants' speech, facial expressions, and tone of voice in real time.

[0036] Step 2: The AI ​​engine uses voice and image recognition technology to quantify the emotions of attendees.

[0037] Step 3: The AI ​​engine uses the analysis results to optimally facilitate the meeting. For example, if it determines that attendees' emotions are leaning negatively, it will pause the meeting and suggest that attendees take a break.

[0038] "Example 2"

[0039] Step 1: The AI ​​engine provides specialized facilitation, such as patent-specific or strategy-specific facilitation.

[0040] Step 2: In patent-specific facilitation, AI automatically asks questions and makes suggestions to promote patent-related discussions.

[0041] Step 3: This reduces the burden on users and improves the quality and effectiveness of meetings.

[0042] "Example 3"

[0043] Step 1: Provide an environment where the AI ​​engine can communicate smoothly even without face-to-face contact.

[0044] Step 2: AI analyzes attendees' emotions and adjusts the meeting progress based on the results.

[0045] Step 3: For example, if the AI ​​determines that attendees' emotions are leaning negatively, it can automatically take appropriate facilitation steps, just like a human would, such as pausing the meeting and suggesting that attendees take a break.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional video conferencing systems have difficulty in properly analyzing the content, facial expressions, and tone of voice of attendees and optimizing the progress of the meeting. Furthermore, there was a lack of means to provide feedback to improve the quality and effectiveness of the meeting, which placed a heavy burden on users and hindered smooth non-face-to-face communication.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes a means for automatically facilitating the video conference using artificial intelligence, a means for analyzing the content of attendees' remarks, facial expressions, and tone of voice and quantifying their emotions, a means for converting voice data into text using speech recognition technology, a means for analyzing facial expressions from video data using image recognition technology, a means for optimal conference facilitation based on the quantified emotional data, and a means for providing feedback based on the progress of the conference and the emotional data. This makes it possible to analyze attendees' emotions in real time and ensure optimal conference progress. Furthermore, by providing feedback to improve the quality and effectiveness of the conference, the burden on users is reduced and smooth communication can be achieved even when not face-to-face.

[0051] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[0052] "Facilitation" is a technique for smoothly progressing meetings and group discussions and enabling participants to exchange opinions effectively.

[0053] "Artificial intelligence" refers to a computer system that mimics human intelligence and has capabilities such as learning, reasoning, and recognition.

[0054] "Speech" refers to the words and opinions expressed orally by participants during a meeting.

[0055] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0056] "Vocal timbre" refers to characteristics such as the quality of the voice, tone, and intonation, and is an element that conveys the speaker's emotions and intentions.

[0057] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[0058] "Quantifying emotions" means quantitatively evaluating emotions and expressing them as numbers.

[0059] "Speech recognition technology" is a technology that converts speech into text, and can automatically record what a speaker says as text.

[0060] "Image recognition technology" is a technology that analyzes image data and recognizes specific patterns and features.

[0061] "Optimal meeting facilitation" is a method for conducting a meeting most effectively, taking into account the circumstances and emotions of the participants.

[0062] "Feedback" refers to evaluations and advice provided based on the progress and results of the meeting.

[0063] "Progress" indicates how the meeting is progressing and its current status.

[0064] "Emotional data" is data that quantifies the emotions of participants and can be used to guide and facilitate meetings.

[0065] This invention relates to a system that uses artificial intelligence to automatically facilitate video conferences. This system analyzes the content of participants' remarks, facial expressions, and tone of voice, and quantifies their emotions to achieve optimal conference facilitation.

[0066] Hardware and software used

[0067] Hardware: Camera, microphone, device (PC, tablet, smartphone)

[0068] Software: Voice recognition technology (e.g., Google® Speech-to-Text), image recognition technology (e.g., OpenCV), AI engine (e.g., TENSORFLOW®)

[0069] System Operation

[0070] Subject: Server

[0071] The server provides a system that automatically facilitates video conferences. This system is equipped with an AI engine that analyzes the content of participants' remarks, facial expressions, and tone of voice. The server uses voice recognition and image recognition technology to quantify the emotions of participants. Based on this quantified emotional data, the server provides optimal conference facilitation. It also provides feedback based on the progress of the conference and the emotional data.

[0072] Subject: Terminal

[0073] The device transmits video and audio from the video conference to the server. The device also uses a camera and microphone to capture the facial expressions and speech of participants. This data is sent to the server in real time and analyzed by an AI engine.

[0074] Subject: User

[0075] Users participate in video conferences and send their comments and facial expressions to the server via their devices. Users simply participate in the conference in a natural way, without any special operations. The AI ​​engine analyzes the content of the user's comments, facial expressions, and tone of voice, providing optimal conference facilitation.

[0076] Specific examples

[0077] Example 1: Analysis of speech during a meeting

[0078] When User A says, "How is the progress on this project?", the server uses voice recognition technology to convert the speech into text. The AI ​​engine then analyzes this text and recognizes that User A is asking a question. The server then facilitates the meeting by encouraging other attendees to provide appropriate answers.

[0079] Example 2: Facial expression analysis

[0080] The device's camera captures User B's facial expression and sends it to the server. The server uses image recognition technology to analyze User B's facial expression and detects that User B is confused. The server pauses the meeting and facilitates by asking User B questions or opinions.

[0081] Prompt Sentence Examples

[0082] "Please describe a system that analyzes what participants say, their facial expressions, and tone of voice during a video conference to provide optimal meeting facilitation. Please include the names of any specific hardware or software."

[0083] In this way, the system's operation can be explained from the perspectives of the server, the device, and the user, and concrete examples are used to deepen understanding. This system can analyze attendees' emotions in real time and ensure optimal meeting progress. It also provides feedback to improve the quality and effectiveness of meetings, reducing the burden on users and enabling smooth communication even when not face-to-face.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1: Start a video conference

[0086] Subject: User

[0087] A user starts a video conference using a terminal. The user starts a conference application and enters the conference ID to join. For example, when a user joins a conference using a conference application, the user enters the conference ID and clicks the "Join" button.

[0088] Input: Meeting ID, user operation

[0089] Output: Start a video conference and prepare to capture video and audio

[0090] Step 2: Capturing Data

[0091] Subject: Terminal

[0092] The device uses a camera and microphone to capture the user's video and audio. The camera captures the user's facial expressions in real time, and the microphone records what the user says. For example, the device's camera captures the user's smile, and the microphone records when the user says "hello."

[0093] Input: User video and audio

[0094] Output: Captured video and audio data

[0095] Step 3: Sending data

[0096] Subject: Terminal

[0097] The device sends the captured video and audio data to the server. The data is encrypted and sent securely to the server. For example, the device sends the captured video and audio data to the server using the HTTPS protocol.

[0098] Input: Captured video and audio data

[0099] Output: Data sent to the server

[0100] Step 4: Analyze the data

[0101] Subject: Server

[0102] The server analyzes the received video and audio data. It uses voice recognition technology to convert the audio data into text and image recognition technology to analyze facial expressions from the video data. For example, the server uses voice recognition technology to convert the audio of "hello" into text and image recognition technology to detect the user's smile.

[0103] Input: Video and audio data sent to the server

[0104] Output: Text data, facial expression analysis results

[0105] Step 5: Quantify emotions

[0106] Subject: Server

[0107] The server quantifies the user's emotions based on the analysis results. The AI ​​engine analyzes changes in voice tone and facial expressions to generate an emotion score. For example, the server quantifies the user's smile as "joy" and assigns an emotion score of 80 / 100.

[0108] Input: Text data, facial expression analysis results

[0109] Output: Sentiment score

[0110] Step 6: Facilitation

[0111] Subject: Server

[0112] The server then optimally facilitates the meeting based on the quantified emotional data. For example, if a user is confused, the server may pause the meeting and prompt the user to ask questions or give their opinions.

[0113] Input: Sentiment score

[0114] Output: Facilitation instructions

[0115] Step 7: Provide feedback

[0116] Subject: Server

[0117] The server provides feedback based on the progress of the meeting and emotional data. For example, after the meeting, it sends a report to attendees about changes in their emotions and the frequency of their comments during the meeting.

[0118] Input: Meeting progress, emotion data

[0119] Output: Feedback report

[0120] In this way, by explaining each processing step specifically and clearly showing the inputs and outputs, the operation of the system can be more clearly understood.

[0121] (Application example 1)

[0122] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] With conventional video conferencing systems, it was difficult to analyze participants' remarks, facial expressions, and tone of voice to quantify their emotions and provide optimal meeting facilitation. Furthermore, while efficient progress is required for factory meetings and discussions, there was a lack of automated means for conducting meetings that took participants' emotions into consideration. This resulted in a decline in the quality and effectiveness of meetings and an increased burden on users.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0125] In this invention, the server includes means for automatically faciliting video conferences using AI, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating the conference, means to be installed in a robot for streamlining in-plant meetings and discussions, and means for optimizing the progress of the conference taking into account the emotions of attendees.This makes it possible to efficiently conduct in-plant meetings and discussions while taking into account the emotions of attendees.

[0126] A "video conference" is a conference in which multiple participants share video and audio in real time via the Internet.

[0127] "Facilitation" refers to methods and techniques for smoothly conducting meetings and discussions.

[0128] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to learn and reason by imitating human intelligence.

[0129] "Speech content" refers to the information and opinions that participants verbally express during a meeting or discussion.

[0130] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0131] "Vocal timbre" refers to the tone and texture of a voice, and is an element that reflects the speaker's emotions and intentions.

[0132] "Quantifying emotions" means analyzing participants' emotional states and expressing them as numerical data.

[0133] "Optimal meeting facilitation" means taking into consideration the content and emotional state of participants and conducting the meeting in the most effective manner.

[0134] A "factory meeting" is a meeting held to discuss matters related to factory operations and production.

[0135] A "meeting" is when multiple people gather to discuss a specific purpose or issue and exchange opinions and share information.

[0136] A "robot" is a mechanical device that operates automatically according to a program to perform specific tasks.

[0137] "Installation" means introducing software or applications into a computer or machine device and making them ready for use.

[0138] "Optimizing the progress of a meeting" means making the necessary adjustments and improvements to ensure that the meeting proceeds efficiently and effectively.

[0139] As an embodiment of this invention, we will explain a "smart conference robot" system for streamlining meetings and discussions within factories. This system uses AI to automatically facilitate video conferences, analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions and provide optimal conference facilitation.

[0140] System configuration

[0141] 1. Hardware:

[0142] Camera: Used to capture attendees' facial expressions in real time.

[0143] Microphone: Used to capture what attendees are saying.

[0144] Robot: A mechanical device that moves around the factory and helps run meetings.

[0145] 2. Software:

[0146] Speech Recognition Library: Use the speech_recognition library to recognize what attendees are saying in real time.

[0147] Emotion analysis model: Using the transformers library pipeline, we analyze attendees' facial expressions and tone of voice to quantify their emotions.

[0148] Image processing library: Uses the cv2 library to acquire images from the camera in real time and perform facial expression analysis.

[0149] System Operation

[0150] 1. Speech Recognition:

[0151] The server converts the audio data acquired from the microphone into text data using the speech_recognition library.

[0152] The converted text data is used as information for optimizing the progress of the conference.

[0153] 2. Emotion analysis:

[0154] The server processes the video data acquired from the camera in real time using the cv2 library and analyzes the facial expressions of the attendees.

[0155] The analyzed facial expression data is used to quantify emotions using a pipeline from the transformers library.

[0156] 3. Optimizing meeting progress:

[0157] The server executes logic to optimize the progress of the conference based on the acquired voice data and emotion data.

[0158] For example, if an attendee says, "There's a problem with this part," the robot will respond, "There's a problem. We'll suggest a solution."

[0159] Also, if the attendee's facial expression is negative, the message "The attendee's emotions are negative. We will adjust the meeting progress." will be displayed.

[0160] Specific examples

[0161] As a concrete example, consider a quality control meeting in a factory. If a participant says, "There is a problem with this part," the robot responds, "A problem has occurred. We will propose a solution." If the participant's facial expression is negative, the robot displays, "The participant's emotions are negative. We will adjust the progress of the meeting."

[0162] Prompt Sentence Examples

[0163] Examples of prompts to input to a generative AI model include:

[0164] If someone says, "We have a problem with this part," please suggest how the meeting should proceed.

[0165] In this way, the smart conference robot system can make meetings within factories more efficient and provide optimal facilitation that takes into account the emotions of attendees.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] The server acquires video data from the camera in real time. The input is video data from the camera, and the output is frame data processed using the image processing library cv2. Specifically, the server captures the video from the camera frame by frame, converts it to grayscale, and performs face detection.

[0169] Step 2:

[0170] The server acquires voice data from the microphone in real time. The input is the voice data from the microphone, and the output is the speech content converted into text data using the speech recognition library, speech_recognition. Specifically, the server records the voice from the microphone and converts it into text using a speech recognition engine.

[0171] Step 3:

[0172] The server analyzes the facial expressions of attendees using the acquired frame data. The input is the frame data obtained in step 1, and the output is emotional data quantified using the emotion analysis model transformers pipeline. Specifically, the server extracts the area where the face was detected, inputs it into the emotion analysis model, and quantifies the emotion.

[0173] Step 4:

[0174] The server analyzes the acquired text data and extracts information necessary for the progress of the meeting. The input is the text data obtained in step 2, and the output is instructions and suggestions for the progress of the meeting. Specifically, the server inputs the text data into a natural language processing engine and extracts important keywords and phrases.

[0175] Step 5:

[0176] The server executes logic to optimize the progress of the meeting based on the emotion data and text data. The input is the emotion data obtained in step 3 and the text data obtained in step 4, and the output is specific actions for the progress of the meeting. As a specific operation, the server integrates the emotion data and text data and generates instructions for adjusting the progress of the meeting.

[0177] Step 6:

[0178] The server sends the generated instructions to the robot to support the progress of the meeting. The input is the instructions for proceeding with the meeting obtained in step 5, and the output is specific actions by the robot. As specific actions, the server sends voice and action instructions to the robot to support the progress of the meeting.

[0179] Step 7:

[0180] The user follows instructions from the robot to conduct the meeting. The input is instructions from the robot, and the output is the progress of the meeting. In concrete terms, the user speaks and acts according to the robot's instructions to ensure the meeting proceeds smoothly.

[0181] Example 2

[0182] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] In conventional video conferencing systems, meeting facilitation is done manually, which places a heavy burden on attendees and can reduce the quality and effectiveness of meetings. Furthermore, patent-related discussions require specialized knowledge, making it difficult to generate appropriate questions and proposals. Furthermore, non-face-to-face communication often does not proceed smoothly, causing meetings to stall.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0185] In this invention, the server includes a means for automatically facilitating video conferences using artificial intelligence, a means for analyzing participants' remarks, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for collecting patent-related data, a means for preprocessing the collected data, a means for generating questions and suggestions using a generative AI model, a means for transmitting the generated questions and suggestions to a terminal, and a means for the terminal to present the results to the user. This reduces the burden on users and improves the quality and effectiveness of conferences. It also promotes specialized discussions related to patents and enables smooth communication even when not face-to-face.

[0186] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[0187] "Facilitation" is the activity of providing support and coordination to ensure the smooth progress of meetings and discussions.

[0188] "Artificial intelligence" is the technology that enables computer systems to learn, reason, and self-correct by imitating human intelligence.

[0189] "Speech content" refers to information and opinions verbally expressed by participants during a meeting or discussion.

[0190] "Facial expressions" are emotions and reactions shown by the movement of facial muscles.

[0191] "Vocal timbre" refers to the characteristics of a voice, such as its quality, tone, and pitch.

[0192] "Quantifying emotions" means expressing emotional states as numerical data.

[0193] A "patent" is a legal right that grants an inventor exclusive use of an invention for a certain period of time.

[0194] "Collecting data" is the act of gathering the necessary information.

[0195] "Preprocessing" is the initial process used to prepare data for easy analysis and use.

[0196] A "generative AI model" is a model that uses artificial intelligence to generate new data and information.

[0197] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific topic.

[0198] A "terminal" is a device that allows a user to input and output information.

[0199] A "user" is someone who uses a system or service.

[0200] "Presenting the results" means showing the generated information or data to the user.

[0201] This invention is a system that uses artificial intelligence to automatically facilitate video conferences. The system has the ability to analyze participants' remarks, facial expressions, and tone of voice to quantify their emotions. It also includes the ability to collect patent-related data, perform preprocessing, and generate questions and suggestions using a generative AI model. A specific embodiment of this system is described below.

[0202] Hardware and Software Configuration

[0203] server

[0204] The server comprises hardware and software to perform the following functions:

[0205] Data collection: The server collects the necessary information from patent-related databases (e.g., patent information databases).

[0206] Data preprocessing: The server uses Python's NLTK library and spaCy to preprocess the collected data.

[0207] Applying a generative AI model: The server uses a pre-trained generative AI model (e.g., GPT-4®) to generate patent-related questions and suggestions.

[0208] Sending results: The server sends the generated questions and suggestions to the device.

[0209] Terminal

[0210] The terminal is a device where the user can input information and display the results sent from the server. The terminal has the following functions:

[0211] Receiving user input: The terminal receives prompts entered by the user.

[0212] Displaying results: The device presents the user with the questions and suggestions sent by the server.

[0213] User

[0214] A user is someone who uses the system to hold a video conference. The user performs the following operations:

[0215] Entering a prompt: The user enters a prompt into the terminal to facilitate discussion related to the patent.

[0216] Specific examples

[0217] For example, consider a case where a user uses this system when holding a meeting regarding a patent.

[0218] Example prompt sentence:

[0219] "I'd like to discuss a new patent application. Please generate questions and suggestions regarding the technical details of the patent and the competitive situation in the market."

[0220] When a user enters this prompt into a terminal, the server collects and preprocesses relevant data from a patent information database. It then uses a generative AI model to generate appropriate questions and suggestions, which it then sends to the terminal. The terminal then presents the generated questions and suggestions to the user. For example, a question such as "How is this technology different from existing patents?" is displayed.

[0221] In this way, users can use the system to efficiently advance patent-related discussions. The system reduces the burden on users and improves the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[0222] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0223] Step 1:

[0224] The user enters a prompt statement.

[0225] The user inputs a prompt sentence into the terminal to promote discussions related to patents. For example, the user inputs, "I would like to hold a discussion about a new patent application. Please generate questions or suggestions about the technical details of the patent and the competitive situation in the market." The input prompt sentence is sent from the terminal to the server.

[0226] Step 2:

[0227] The server collects the data.

[0228] The server receives the prompt sent by the user and collects data related to patents. Specifically, it retrieves relevant patent information from a patent information database. For example, it calls the API of a patent information database to collect patent information related to "new technology." The input is the prompt, and the output is the collected patent information.

[0229] Step 3:

[0230] The server preprocesses the data.

[0231] The server preprocesses the collected patent information. It uses Python's NLTK library and spaCy to analyze the text data and extract important keywords and phrases. For example, it extracts keywords such as "technical details" and "market competitive situation." The input is the collected patent information, and the output is the preprocessed data.

[0232] Step 4:

[0233] The server applies the generative AI model.

[0234] The server generates questions and suggestions using a generative AI model (e.g., GPT-4) based on the preprocessed data. The generative AI model generates appropriate questions and suggestions based on the extracted keywords. For example, it generates questions such as, "How is this technology different from existing patents?" The input is the preprocessed data, and the output is the generated questions and suggestions.

[0235] Step 5:

[0236] The server transmits the generated results to the terminal.

[0237] The server sends the generated questions and suggestions to the terminal. The generated questions and suggestions are sent to the terminal in JSON format. For example, they are sent in the format {"question": "How is this technology different from existing patents?"}. The input is the generated questions and suggestions, and the output is the data sent to the terminal.

[0238] Step 6:

[0239] The terminal presents the results to the user.

[0240] The terminal presents the questions and suggestions received from the server to the user. The user can then advance the discussion based on the presented questions and suggestions. For example, the terminal screen might display "How is this technology different from existing patents?" The input is the data sent from the server, and the output is the questions and suggestions presented to the user.

[0241] (Application example 2)

[0242] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0243] In conventional video conferencing systems, meeting facilitation is done manually, which places a heavy burden on attendees and can reduce the quality and effectiveness of meetings. Furthermore, patent-related discussions and technical debates require specialized knowledge, making it difficult to ask appropriate questions or make appropriate suggestions. Furthermore, the inability to smoothly communicate without face-to-face contact is also an issue.

[0244] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0245] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing attendees' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for generating questions and suggestions to promote patent-related discussions, and means for generating prompt sentences based on the conference topic using a generative AI model.This improves the quality and effectiveness of conferences, efficiently supports patent-related discussions and technical debates, and enables smooth communication even when not face-to-face.

[0246] A "video conference" is a conference in which multiple participants share video and audio in real time via the Internet.

[0247] "Facilitation" is the activity of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[0248] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0249] "Speech content" refers to information or opinions expressed orally by participants in a meeting or discussion.

[0250] "Facial expressions" refer to emotions and reactions expressed through the movement of facial muscles.

[0251] "Vocal timbre" refers to characteristics such as the quality of a voice, tone, and intonation.

[0252] "Quantifying emotions" means expressing emotional states as numerical data.

[0253] A "patent" is an exclusive right granted to an invention.

[0254] "Facilitating discussion" means supporting participants to actively exchange opinions and engage in lively debate.

[0255] "Generating questions and suggestions" refers to automatically creating appropriate questions and suggestions based on a specific topic.

[0256] A "generative AI model" is an artificial intelligence model that generates new data or information based on input data.

[0257] A "prompt" is an instruction or question that is input into a generative AI model.

[0258] "Inside a factory" refers to the inside of a facility where production activities in the manufacturing industry take place.

[0259] A "technical discussion" is an exchange of opinions and discussion between experts on technical matters.

[0260] A system for carrying out the present invention includes a program for automatically facilitating a video conference. The system is realized using the following hardware and software.

[0261] Hardware

[0262] Server: A server with high-performance computing power is required to run generative AI models and analyze data quickly.

[0263] Device: You will need a device (computer, tablet, smartphone, etc.) to participate in the video conference.

[0264] Camera and microphone: A camera and microphone are required to capture what participants are saying, their facial expressions, and tone of voice.

[0265] software

[0266] Generative AI model: Uses OpenAI® API to generate prompts based on the meeting topic.

[0267] Data analysis software: Software is needed to analyze what attendees say, their facial expressions, and tone of voice, and to quantify their emotions.

[0268] Videoconferencing software: Software for conducting video conferences (e.g., Zoom or MICROSOFT® TEAMS®) is required.

[0269] Data processing and calculation

[0270] The server analyzes participants' comments, facial expressions, and tone of voice in real time during video conferences to quantify their emotions. This allows the system to grasp the progress of the conference and participants' reactions. Furthermore, it uses a generative AI model to generate prompts based on the conference topic, automatically asking questions and making suggestions to promote patent-related discussions.

[0271] Specific examples

[0272] For example, if a meeting is being held regarding the patenting of a new manufacturing process, the server might generate a prompt like this:

[0273] "What technical features should be highlighted to patent this manufacturing process?"

[0274] "What are the advantages of this process compared to competitor patents?"

[0275] This allows attendees to efficiently advance discussions, improving the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[0276] In this way, the system for implementing the invention can automate the facilitation of video conferences to support patent-related discussions and technical debates.

[0277] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0278] Step 1:

[0279] At the start of a video conference, the server activates the camera and microphone to capture what participants are saying, their facial expressions, and tone of voice. As input, it receives real-time data from the camera and microphone, and as output, it sends this data to software for analysis.

[0280] Step 2:

[0281] The server uses analysis software to analyze the acquired data on speech content, facial expressions, and tone of voice, and quantifies emotions. It receives real-time data from the camera and microphone as input, and generates quantified emotional data as output. Specifically, it converts speech content into text using voice recognition technology, and analyzes facial muscle movements using facial expression recognition technology.

[0282] Step 3:

[0283] Based on the analysis results, the server uses a generative AI model to generate a prompt based on the meeting topic. It receives quantified emotion data and the meeting topic as input, and generates a prompt as output. Specifically, it generates a prompt using OpenAI's API.

[0284] Step 4:

[0285] The server sends the generated prompt sentence to the video conferencing software and presents it to the attendees. The server receives the generated prompt sentence as input and displays the prompt sentence on the video conferencing software as output. As a specific operation, the server displays the prompt sentence using a chat function or a screen sharing function of the video conferencing software.

[0286] Step 5:

[0287] The user advances the discussion based on the presented prompt sentence. The system receives the prompt sentence as input and generates the content of the discussion as output. As a specific action, the user expresses their opinion or question in response to the prompt sentence.

[0288] Step 6:

[0289] The server monitors the progress of the discussion and generates additional prompts as needed. It receives the content and emotion data of the discussion as input and generates additional prompts as output. Specifically, it uses the generative AI model again to generate new prompts.

[0290] Step 7:

[0291] At the end of the meeting, the server saves all data and generates a report of the meeting. It receives as input all data acquired during the meeting and generates as output a report of the meeting. Specific operations include saving the data in a database and creating a report using report generation software.

[0292] Example 3

[0293] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] With conventional video conferencing systems, it was difficult to analyze attendees' emotions in real time and adjust the progress of the meeting appropriately. This resulted in problems such as a decline in the quality and effectiveness of the meeting and an increase in the burden on users. In addition, non-face-to-face communication was often not possible, which caused the progress of the meeting to be delayed.

[0295] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for automatically faciliting the conference using artificial intelligence, means for analyzing the content of attendees' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for collecting audio data of the conference, means for transmitting the collected audio data to the server, means for the server to perform emotion analysis of the audio data, means for adjusting the progress of the conference based on the analysis results, and means for suggesting appropriate actions to the user. This improves the quality and effectiveness of the conference, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0296] "Means for automated meeting facilitation using artificial intelligence" refers to a function that uses artificial intelligence to automatically progress and coordinate meetings.

[0297] "Means of analyzing the speech content, facial expressions, and tone of voice of attendees and quantifying their emotions" is a function that analyzes the speech, facial expressions, and tone of voice of attendees and expresses their emotions as a number.

[0298] "Means for optimal meeting facilitation" is a function that makes adjustments to optimize the progress of meetings based on analyzed emotional data.

[0299] The "means for collecting conference audio data" is a function for collecting audio during a conference in real time.

[0300] The "means for transmitting collected voice data to a server" is a function for transmitting collected voice data to a server via the Internet.

[0301] The "means for the server to analyze the emotions of the voice data" is a function for the server to analyze the voice data received and identify the emotions of the attendees.

[0302] "Means for adjusting the progress of the meeting based on the analysis results" is a function for appropriately adjusting the progress of the meeting based on the results of emotion analysis.

[0303] The "means for suggesting appropriate actions to the user" is a function for suggesting appropriate actions to the user based on the analysis results.

[0304] This invention relates to a system that uses artificial intelligence to automatically facilitate meetings. This system analyzes the content, facial expressions, and tone of voice of attendees and quantifies their emotions to enable optimal meeting facilitation.

[0305] 1. Generate the system program

[0306] The system was developed using Python, with the main libraries being "Hugging Face Transformers" for emotion analysis and "Google Calendar API" for meeting management.

[0307] 2. Explain the program's processing in natural language

[0308] When a user starts a meeting, the device collects audio data in real time. The collected audio data is sent to a server, where it undergoes emotion analysis using Hugging Face Transformers. The analysis results are classified into three categories: positive, negative, and neutral.

[0309] The server adjusts the progress of the meeting based on the analysis results. For example, if it determines that attendees' emotions are leaning negatively, the server will pause the meeting through the Google Calendar API and display a message on the device suggesting a break.

[0310] 3. Add examples to your description

[0311] As a concrete example, the following scenario can be considered.

[0312] scenario:

[0313] A user starts an online meeting. During the meeting, one of the attendees repeatedly makes negative comments. The system performs sentiment analysis on the comments and determines that the comments have a strong negative sentiment. The server pauses the meeting and displays a message on the device asking, "Suggest taking a break?"

[0314] Example prompt sentence:

[0315] "Develop a system that analyzes attendees' emotions in real time during online meetings and suggests pausing the meeting and taking a break if negative emotions are present."

[0316] This system provides an environment in which smooth communication can be achieved even without face-to-face contact. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0317] Step 1:

[0318] A user starts a conference. The user launches the conference application and clicks the "Start Conference" button. This causes the system to start collecting voice data. The input is the user's operation, and the output is the conference start signal.

[0319] Step 2:

[0320] The device collects the audio data of the meeting. The device picks up the audio during the meeting with a microphone and converts it into digital audio data in real time. For example, when a user says "Hello, everyone," the audio is immediately collected as digital data. The input is the audio during the meeting, and the output is digital audio data.

[0321] Step 3:

[0322] The terminal sends the collected voice data to the server. The terminal sends the collected voice data to the server using the HTTPS protocol. The data is AES encrypted and transferred securely. The input is digital voice data, and the output is a transmission completion signal to the server.

[0323] Step 4:

[0324] The server performs emotion analysis on the voice data. The server inputs the received voice data into the Hugging Face Transformers emotion analysis model. For example, a statement such as "This project might not work out" is analyzed as negative. The input is digital voice data, and the output is the emotion analysis result.

[0325] Step 5:

[0326] The server adjusts the progress of the meeting based on the analysis results. If the server determines that the result of the sentiment analysis is negative, it issues an instruction to pause the progress of the meeting using the Google Calendar API. The input is the sentiment analysis result, and the output is an instruction to adjust the progress of the meeting.

[0327] Step 6:

[0328] The terminal suggests appropriate actions to the user. The terminal receives instructions from the server and displays a message to the user saying, "Attendees' emotions are negative. Would you like to suggest taking a break?" The user adjusts the progress of the meeting by selecting "Yes" or "No." The input is an instruction from the server, and the output is a suggested message to the user.

[0329] In this way, the system provides an environment in which smooth communication can be carried out even without face-to-face contact.

[0330] (Application example 3)

[0331] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] In conventional video conferencing systems and work environments, it was difficult to properly analyze the emotions of attendees and workers and optimize meeting progress and work efficiency. Furthermore, when emotions turned negative, it was not possible to respond appropriately, which led to problems with reduced meeting quality and work safety. This increased the burden on users and hindered smooth non-face-to-face communication.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0334] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time to improve work efficiency and safety, and means for pausing work and suggesting a break when emotions become negative.This optimizes the progress of the conference and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0335] A "video conference" is a conference in which multiple participants share video and audio via the Internet.

[0336] "Facilitation" means providing support and coordination to ensure that meetings and work progress smoothly.

[0337] "AI" stands for artificial intelligence, a technology that allows machines to imitate human intelligence and learn and reason.

[0338] "Speech" refers to information and opinions that participants verbally express during a meeting or task.

[0339] "Facial expression" refers to emotions and intentions shown by facial muscle movements.

[0340] "Tone" refers to the quality and tone of a voice, and reflects the speaker's emotions and state.

[0341] "Quantifying emotions" means expressing emotions as quantitative data.

[0342] A "worker" is someone who performs a specific task in a factory, office, etc.

[0343] "Real time" means dealing with what's happening right now.

[0344] "Work efficiency" refers to the ability to minimize waste of time and effort while performing work and achieve maximum results.

[0345] "Safety" means a state in which work or activities can be carried out without risk and with peace of mind.

[0346] "Suggesting a break" means temporarily pausing work or the progress of a meeting to encourage participants to take a rest.

[0347] A "system" is a mechanism in which multiple elements work together to perform a specific function.

[0348] A system for implementing this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating meetings, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for pausing work and suggesting a break when emotions become negative.

[0349] System Program

[0350] The program of this system performs the following processing.

[0351] 1. Hardware: The system uses a camera (e.g., Logitech C920) to capture the faces of attendees and workers.

[0352] 2. Software: The system uses software such as OpenCV, Keras, and TensorFlow to process the captured images and analyze emotions.

[0353] 3. Data processing: The captured image is converted to grayscale and face detection is performed. The detected face regions are preprocessed to be input into the emotion analysis model.

[0354] 4. Data computation: Predict emotions from preprocessed facial images using an emotion analysis model.

[0355] Specific examples of processing

[0356] For example, if a worker in a factory feels stressed, the system will analyze their emotions in real time and suggest pausing work and taking a break, improving work efficiency and safety while reducing the burden on the worker.

[0357] Prompt Sentence Examples

[0358] An example of a prompt to input to a generative AI model is as follows:

[0359] Develop a robot assistant application in a factory that analyzes workers' emotions in real time to improve work efficiency and safety. Include a function that allows the robot to pause work and suggest a break if the worker is feeling stressed.

[0360] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0361] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0362] Step 1:

[0363] The server uses a camera to capture the faces of attendees and workers. The input is video data from the camera, and the output is the captured image data. Specifically, the camera acquires video in real time and captures the video frame by frame.

[0364] Step 2:

[0365] The server converts the captured image data to grayscale. The input is the captured image data, and the output is grayscale image data. Specifically, it uses OpenCV to convert the image to grayscale.

[0366] Step 3:

[0367] The server detects faces from grayscale images. The input is grayscale image data, and the output is coordinate data of the detected face area. Specifically, the server identifies the face area using the OpenCV face detection algorithm.

[0368] Step 4:

[0369] The server preprocesses the detected face region to input it into the emotion analysis model. The input is the coordinate data and grayscale image data of the face region, and the output is the preprocessed face image data. Specifically, the server cuts out the face region, resizes it to 48x48 pixels, and normalizes it.

[0370] Step 5:

[0371] The server inputs the preprocessed facial image data into the emotion analysis model to predict emotions. The input is the preprocessed facial image data, and the output is the emotion prediction result. Specifically, Keras is used to input data into the emotion analysis model and obtain the prediction result.

[0372] Step 6:

[0373] The server determines whether the predicted emotion is negative. The input is the emotion prediction result, and the output is the negative emotion judgment result. Specifically, the server analyzes the prediction result and checks whether it contains negative emotions (e.g., anger, sadness, fear).

[0374] Step 7:

[0375] If a negative emotion is detected, the server pauses the work and suggests taking a break. The input is the negative emotion judgment result, and the output is a notification suggesting a break. Specifically, the system pauses the work and sends a notification to the user to take a break.

[0376] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0377] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0378] "Example 1"

[0379] In one embodiment of the present invention, the emotion engine recognizes a user's emotions and quantifies them. Specifically, the emotion is estimated from the user's remarks, tone of voice, facial expressions, etc., and quantified. This quantified emotion data is used for meeting facilitation. For example, if the AI ​​determines that the emotions of all participants are leaning negatively, it will pause the meeting and take action such as suggesting a break.

[0380] "Example 2"

[0381] The emotion engine also detects changes in the user's emotions in real time and adjusts the meeting facilitation accordingly. For example, if it senses that a participant is showing anger or dissatisfaction, the AI ​​will take action such as offering that participant an opportunity to speak or conveying that emotion to other participants. This helps the meeting proceed smoothly and improves the satisfaction of all participants.

[0382] "Example 3"

[0383] Furthermore, the emotion engine predicts changes in the user's emotions and optimizes meeting facilitation based on the predictions. For example, if it predicts that a certain participant is likely to be dissatisfied with the progress of the meeting, the AI ​​will take action such as providing that participant with an opportunity to speak in advance or adjusting the progress of the meeting. This will help the meeting proceed more smoothly and further improve the satisfaction of all participants.

[0384] The processing flow of each embodiment will be described below.

[0385] "Example 1"

[0386] Step 1: Estimate the user's emotions from their speech, tone of voice, facial expressions, etc.

[0387] Step 2: Quantify the estimated emotions.

[0388] Step 3: Use quantified emotional data to facilitate meetings.

[0389] Step 4: If the overall sentiment of the participants is judged to be negative, the AI ​​will take action such as pausing the meeting and suggesting a break.

[0390] "Example 2"

[0391] Step 1: The emotion engine detects changes in the user's emotions in real time.

[0392] Step 2: Adjust your meeting facilitation as emotions change.

[0393] Step 3: If the AI ​​senses that a participant is showing anger or dissatisfaction, it takes action such as offering that participant an opportunity to speak or conveying their feelings to other participants.

[0394] "Example 3"

[0395] Step 1: The emotion engine predicts changes in the user's emotions.

[0396] Step 2: Optimize meeting facilitation based on predictions.

[0397] Step 3: If it predicts that a participant is likely to be dissatisfied with the progress of the meeting, the AI ​​will take action such as providing that participant with an opportunity to speak in advance or adjusting the progress of the meeting.

[0398] Example 1

[0399] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0400] With conventional video conferencing systems, it was difficult to analyze attendees' emotions and comments in real time and optimize the progress of the meeting. There was also a lack of means to provide professional facilitation to improve the quality and effectiveness of meetings. Furthermore, there was no environment in place to reduce the burden on users and ensure smooth communication in non-face-to-face communication.

[0401] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0402] In this invention, the server includes means for capturing users' speech content and facial expressions in real time using a high-resolution camera and microphone, means for transmitting the captured audio data and video data to the server, means for converting the audio data into text using voice recognition software, means for analyzing the user's facial expressions from the video data using image recognition software, means for quantifying the user's emotions from the text data and facial expression data using an emotion analysis engine, and means for optimizing the progress of the conference based on the quantified emotional data. This makes it possible to analyze attendees' emotions and speech content in real time and provide optimal conference facilitation.

[0403] A "high-resolution camera" is a camera capable of capturing high-resolution video to capture the user's face and facial expressions in detail.

[0404] A "microphone" is a voice input device for accurately recording what a user says.

[0405] "Voice data" refers to voice information including the content of a user's speech acquired through a microphone.

[0406] "Video data" refers to video information including the user's face and facial expressions captured through a high-resolution camera.

[0407] A "server" is a computer system for receiving and analyzing audio and video data.

[0408] "Speech recognition software" is software that analyzes voice data and converts it into corresponding text data.

[0409] "Text data" is text information generated from voice data by voice recognition software.

[0410] "Image recognition software" is software that analyzes video data and recognizes the user's facial expressions.

[0411] An "emotion analysis engine" is an analysis system that estimates and quantifies a user's emotions based on text data and facial expression data.

[0412] "Emotion data" is user emotional information quantified by an emotion analysis engine.

[0413] "Means for optimizing the progress of meetings" refers to means for adjusting the progress of meetings and providing optimal facilitation based on quantified emotional data.

[0414] This invention relates to a system that automatically facilitates video conferences using AI. This system analyzes the content of participants' remarks, facial expressions, and tone of voice, and quantifies their emotions to provide optimal conference facilitation.

[0415] Hardware and software used

[0416] Hardware: high-resolution camera, microphone, server

[0417] Software: speech recognition software (e.g., Google Speech-to-Text), image recognition software (e.g., OpenCV), sentiment analysis engines (e.g., IBM Watson®)

[0418] System Overview

[0419] 1. Acquiring user utterances and facial expressions

[0420] The device uses a high-resolution camera and microphone to capture what the user is saying and their facial expressions in real time.

[0421] Specifically, a camera captures the user's face and a microphone records the user's voice.

[0422] For example, if a user frowns while saying "this project is difficult," the camera and microphone will capture the moment.

[0423] 2. Data transmission

[0424] The terminal transmits the captured audio and video data to the server.

[0425] Specifically, the terminal divides the data into packets and sends them to a server over the Internet.

[0426] For example, data on the user's speech and facial expressions is sent to the server in real time.

[0427] 3. Voice and Image Recognition

[0428] The server uses voice recognition software to convert the voice data into text.

[0429] At the same time, the server uses image recognition software to analyze the user's facial expressions from the video data.

[0430] Specifically, speech recognition software analyzes speech waveforms and generates corresponding text, while image recognition software detects facial features and analyzes facial expressions.

[0431] For example, a statement such as "This project is difficult" is converted into text, and a frown is analyzed as "confusion."

[0432] 4. Quantifying emotions

[0433] The server uses an emotion analysis engine to estimate the user's emotions from text data and facial expression data and convert them into numerical values.

[0434] Specifically, the sentiment analysis engine takes text and facial expression data as input and generates an emotion score.

[0435] For example, a "confused" facial expression and a "difficult" statement will result in a high negative emotion score.

[0436] 5. Meeting Facilitation

[0437] The server optimizes the progress of the meeting based on the quantified emotional data.

[0438] Specifically, the server analyzes the emotion scores and, if necessary, pauses the meeting and suggests a break.

[0439] For example, if the emotion score of all participants is judged to be negative, the server will suggest, "Let's take a break."

[0440] Specific examples

[0441] For example, the following scenario is possible:

[0442] Scenario: During a meeting, many participants look tired and their comments become increasingly negative.

[0443] process:

[0444] The device's camera and microphone capture participants' facial expressions and speech.

[0445] The server uses voice recognition software to convert what is said into text and uses image recognition software to analyze facial expressions.

[0446] The emotion analysis engine quantifies emotions from text and facial expression data and determines that negative emotions are prevalent.

[0447] The server pauses the conference and suggests a break.

[0448] Prompt Sentence Examples

[0449] An example of a prompt sentence to input to the generative AI model is as follows:

[0450] "If many of the participants look tired and their comments become increasingly negative during the meeting, please suggest how to proceed."

[0451] By inputting this prompt into a generative AI model, the AI ​​will suggest appropriate ways to facilitate the meeting.

[0452] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0453] Step 1:

[0454] The user speaks and shows facial expressions.

[0455] Input: User speech and facial expressions.

[0456] Specific behavior: A user says, "This project is difficult," and frowns.

[0457] Step 2:

[0458] The device uses a high-resolution camera and microphone to capture what the user is saying and their facial expressions in real time.

[0459] Input: User speech and facial expressions.

[0460] Output: Audio and video data.

[0461] What it does: The camera captures the user's face and the microphone records the user's voice.

[0462] Step 3:

[0463] The terminal transmits the captured audio and video data to the server.

[0464] Input: Audio and video data.

[0465] Output: Audio and video data sent to the server.

[0466] What it does: The device breaks the data into packets and sends them over the Internet to a server.

[0467] Step 4:

[0468] The server uses voice recognition software to convert the voice data into text.

[0469] Input: Audio data.

[0470] Output: Text data.

[0471] What it does: Speech recognition software analyzes the audio waveform and generates corresponding text.

[0472] Step 5:

[0473] The server uses image recognition software to analyze the user's facial expressions from the video data.

[0474] Input: Video data.

[0475] Output: Facial expression data.

[0476] What it does: Image recognition software detects facial features and analyzes facial expressions.

[0477] Step 6:

[0478] The server uses an emotion analysis engine to quantify the user's emotions from text data and facial expression data.

[0479] Input: Text data and facial expression data.

[0480] Output: Emotion data.

[0481] What it does: The sentiment analysis engine takes text and facial expression data as input and generates an emotion score.

[0482] Step 7:

[0483] The server optimizes the progress of the meeting based on the quantified emotional data.

[0484] Input: Emotion data.

[0485] Output: Instructions for running the meeting.

[0486] What it does: The server analyzes the emotion scores and pauses the meeting and suggests breaks as needed.

[0487] (Application example 1)

[0488] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0489] Conventional video conferencing systems and in-factory work support systems have had difficulty in properly analyzing the emotions of participants and workers and providing optimal facilitation and work support based on that analysis. Particularly during long work sessions and meetings, stress and fatigue accumulate among participants and workers, resulting in reduced efficiency and safety. This has created a demand for new methods to improve the quality of meetings and the efficiency and safety of work.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0491] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time and providing support for improving work efficiency and safety, means for suggesting breaks based on the emotion recognition results, and means for notifying the suggestion by voice using a speech synthesis engine, thereby making it possible to improve the efficiency and safety of meetings and work.

[0492] "Video conference facilitation" is the act of coordinating the proceedings and participants' comments during a video conference, and helping to achieve the meeting's objectives.

[0493] "AI automated means" refers to methods or devices that use artificial intelligence to automatically perform specific tasks.

[0494] "Means for analyzing the content of speech, facial expressions, and tone of voice of attendees and quantifying their emotions" refers to a method or device that uses voice recognition technology or image recognition technology to analyze the content of speech, facial expressions, and tone of voice of attendees and express the results as numerical values.

[0495] "Means for optimally facilitating a meeting" refers to a method or device for optimizing the progress of a meeting based on analyzed data.

[0496] "Means for analyzing worker emotions in real time and providing support to improve work efficiency and safety" refers to a method or device for analyzing worker emotions in real time and improving work efficiency and safety based on the results.

[0497] The "means for suggesting a break based on the result of emotion recognition" refers to a method or device for suggesting a break to workers or attendees based on the result of emotion recognition.

[0498] The "means for notifying a proposal by voice using a voice synthesis engine" refers to a method or device for notifying the proposal content by voice using voice synthesis technology.

[0499] The following system configuration is used as an embodiment of the present invention: The server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time and providing support to improve work efficiency and safety, means for suggesting breaks based on the emotion recognition results, and means for notifying suggestions by voice using a speech synthesis engine.

[0500] Hardware and software used

[0501] Hardware: Webcam, microphone

[0502] Software: OpenCV, Keras, pyttsx3

[0503] Data processing and calculation

[0504] The server captures video and audio in real time using a webcam and microphone. The captured video data undergoes facial recognition using OpenCV, and facial regions are extracted. The extracted facial regions are input into a Keras emotion recognition model, and emotions are quantified. Audio data is analyzed in the same way, and emotions are estimated from the tone of voice.

[0505] Based on the analyzed emotion data, the server will optimally facilitate the meeting. For example, if the emotions of all attendees are negative, the server will pause the meeting and suggest a break. Also, if a worker's emotions indicate stress or fatigue, the server will suggest a break. These suggestions are announced aloud using a speech synthesis engine (pyttsx3).

[0506] Specific examples

[0507] For example, consider a scenario in which an emotion recognition system detects the stress level of a worker who has been working for a long time in a factory and suggests that the worker take a break. In this case, the server uses a webcam and microphone to analyze the worker's facial expressions and tone of voice, and if it determines that the worker's stress level is high, it will issue a voice message saying, "Please take a break."

[0508] Example prompts to input to the generative AI model:

[0509] "Design a system that analyzes the emotions of workers in a factory in real time and suggests taking a break if stress or fatigue increases. Specifically, the system analyzes the facial expressions and tone of voice of workers and quantifies their emotions."

[0510] In this way, it is possible to improve the efficiency and safety of meetings and work.

[0511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0512] Step 1:

[0513] The server captures video and audio in real time using a webcam and microphone. The input is video data from the webcam and audio data from the microphone. The output is the captured raw video and audio data.

[0514] Step 2:

[0515] The server performs face recognition on the acquired video data using OpenCV. The input is the video data acquired in step 1. The output is image data with the face area cut out. Specifically, the position of the face is detected from the video data and that area is cut out.

[0516] Step 3:

[0517] The server inputs the extracted face region into the Keras emotion recognition model and quantifies the emotion. The input is the image data of the face extracted in step 2. The output is numerical data indicating the emotion. Specifically, the face image data is preprocessed and input into the emotion recognition model to predict the emotion.

[0518] Step 4:

[0519] The server analyzes the acquired voice data and estimates emotions from the tone of voice. The input is the voice data acquired in step 1. The output is numerical data indicating emotions based on the tone of voice. Specifically, the server preprocesses the voice data and extracts voice features to estimate emotions.

[0520] Step 5:

[0521] The server integrates the emotion data obtained in steps 3 and 4 to perform a comprehensive emotion evaluation. The input is facial emotion data and tone of voice emotion data. The output is numerical data indicating the comprehensive emotion evaluation. Specifically, the facial emotion data and tone of voice emotion data are integrated to calculate a comprehensive emotion score.

[0522] Step 6:

[0523] The server optimally facilitates the meeting and provides work support based on the overall emotional evaluation. The input is the overall emotional evaluation data obtained in step 5. The output is a specific action (e.g., suggesting a break). Specifically, if the emotional evaluation is negative, the server pauses the meeting and suggests a break.

[0524] Step 7:

[0525] The server uses a speech synthesis engine (pyttsx3) to notify the proposed content by voice. The input is the action determined in step 6. The output is a voice notification. Specifically, the break proposal is notified to attendees and workers by voice.

[0526] In this way, the server can analyze emotions in real time and provide optimal meeting facilitation and work support.

[0527] Example 2

[0528] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0529] In conventional video conferencing systems, meeting progress and facilitation are done manually, placing a heavy burden on attendees and reducing the quality and effectiveness of the meeting. It is also difficult to grasp attendees' emotions in real time and respond accordingly, which can impede smooth communication. Furthermore, when conducting specialized discussions, such as those focused on patents or strategies, it is difficult to generate appropriate questions and suggestions.

[0530] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for automatically facilitating the video conference using artificial intelligence, a means for analyzing the content of attendees' remarks, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for generating relevant questions and suggestions based on the conference theme and purpose, and a means for adjusting the progress of the conference based on emotion data. This reduces the burden on attendees and improves the quality and effectiveness of the conference. Furthermore, smooth communication can be achieved through real-time responses based on emotion data. Furthermore, even in specialized discussions such as those focused on patents or strategies, the depth and efficiency of discussions can be improved by automatically generating appropriate questions and suggestions.

[0531] A "video conference" is a type of conference in which multiple participants communicate in real time using audio and video over the Internet.

[0532] "Facilitation" is the process of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[0533] "Artificial intelligence" refers to computer systems that mimic human intelligence and can learn, reason, and solve problems.

[0534] "Speech" refers to information and opinions verbally expressed by participants during a meeting.

[0535] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0536] "Vocal timbre" refers to characteristics of speech, such as tone, pitch, and rhythm, and is an element that conveys the speaker's emotions and intentions.

[0537] "Quantifying emotions" means expressing analyzed emotional data as quantitative numbers.

[0538] "Optimal meeting facilitation" means conducting and adjusting the meeting in the most effective manner depending on the purpose and situation of the meeting.

[0539] A "conference theme" is the main topic or issue to be discussed at a conference.

[0540] "Purpose" refers to the specific goal or outcome that the meeting aims to achieve.

[0541] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on the theme and purpose of the meeting.

[0542] "Emotional data" refers to analytical results that indicate the emotional state of participants.

[0543] "Coordinating the progress of a meeting" means properly managing the flow and progress of the meeting and making adjustments to ensure that it proceeds smoothly.

[0544] "Patent-specific" is a conference format specialized for discussing and examining patent-related matters.

[0545] "Strategy-focused" is a meeting format specialized for discussing and examining strategy-related matters.

[0546] MODE FOR CARRYING OUT THE INVENTION

[0547] This invention is a system that uses artificial intelligence to automatically facilitate video conferences, and has the ability to analyze attendees' comments, facial expressions, and tone of voice to quantify their emotions. It also has the ability to generate relevant questions and suggestions based on the theme and purpose of the conference, and adjust the progress of the conference based on emotional data.

[0548] System configuration

[0549] This system consists of three main components: a server, a terminal, and a user.

[0550] server

[0551] The server uses a generative AI model to generate questions and suggestions based on the meeting's theme and purpose. For example, OpenAI's GPT-4 is used as a generative AI model. The server receives input from the user, generates appropriate questions and suggestions based on that information, and sends them to the device. The server also receives emotion data sent from the device and determines actions to take to adjust the progress of the meeting.

[0552] Terminal

[0553] The device uses a camera and microphone to detect the user's emotions in real time. The emotion engine uses, for example, the Emotion API from Microsoft's Azure (registered trademark) Cognitive Services. The device captures the user's facial expressions through the camera and records the tone of voice through the microphone. This data is sent to the emotion engine, which analyzes the user's emotions in real time. The device receives instructions from the server and adjusts the progress of the meeting.

[0554] User

[0555] Users use their devices to input the topic and purpose of the meeting, for example, "I want to discuss new patent ideas." If a user expresses anger or frustration during the meeting, the device sends that information to the server, which then decides what action to take to adjust the progress of the meeting.

[0556] Specific examples

[0557] As a concrete example, consider the case of patent-specific facilitation. When a user inputs, "I would like to discuss a new patent idea," the server uses a generative AI model to automatically generate questions such as, "What is the technical novelty of this idea?" and "How is it different from existing patents?"

[0558] As a concrete example of an emotion engine, consider the case where a participant expresses dissatisfaction during a meeting. The device analyzes the participant's facial expression and tone of voice to detect dissatisfaction. The server receives this information and instructs the device to provide an opportunity for the participant to speak, such as, "Mr. / Ms. XX, do you have any comments?"

[0559] Prompt Sentence Examples

[0560] "I'd like to discuss new patent ideas."

[0561] "Generate questions about the novelty of this technology."

[0562] "Please suggest appropriate responses to participants who express dissatisfaction during a meeting."

[0563] In this way, the system can reduce the burden on users and improve the quality and effectiveness of meetings.

[0564] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0565] Step 1:

[0566] The user inputs the theme and purpose of the meeting. The user uses the keyboard on the device to type, "I would like to discuss new patent ideas." This input data is sent from the device to the server.

[0567] Step 2:

[0568] The server uses a generative AI model to generate questions and suggestions related to the conference theme. The server inputs the theme and purpose received from the user into the generative AI model, generating questions such as "What is the technical novelty of this idea?" and "How is it different from existing patents?" The generated questions are sent from the server to the terminal.

[0569] Step 3:

[0570] The device detects the user's emotions in real time. The device captures the user's facial expressions through the camera and records the tone of voice through the microphone. This data is sent to the emotion engine, which analyzes the user's emotions in real time. For example, if the user frowns, the emotion of anger is detected. This emotion data is sent from the device to the server.

[0571] Step 4:

[0572] The server receives the emotion data and determines an action to adjust the progress of the conference. The server receives the emotion data sent from the terminal and determines an action to adjust the progress of the conference based on that data. For example, if a user expresses dissatisfaction, the server instructs the user to be given an opportunity to speak. This instruction is sent from the server to the terminal.

[0573] Step 5:

[0574] The terminal follows instructions from the server and adjusts the progress of the conference. The terminal receives instructions from the server and displays a message to the user asking, "Mr. / Ms. XX, do you have any comments?" When a user begins to speak, the terminal sends the content to the server, ensuring the conference proceeds smoothly.

[0575] (Application example 2)

[0576] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0577] With conventional video conferencing systems, it was difficult to detect participants' emotions in real time and adjust the progress of the meeting based on that. Furthermore, there was a lack of means to provide specialized facilitation for patents or strategies, making it difficult to improve the quality and effectiveness of meetings. Furthermore, there was a need for a way to properly manage participants' emotions and promote smooth communication in factory meetings and brainstorming sessions.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0579] In this invention, the server includes a means for automatically facilitating video conferences using AI, a means for analyzing participants' comments, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for supporting factory meetings and brainstorming sessions, and a means for detecting participants' emotions in real time using an emotion engine and adjusting the progress of the conference. This makes it possible to detect participants' emotions in real time and adjust the progress of the conference based on that information, providing specialized facilitation for patents and strategies. It also makes it possible to appropriately manage participants' emotions in factory meetings and brainstorming sessions, promoting smooth communication.

[0580] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[0581] "Facilitation" is the activity of helping meetings and group discussions proceed smoothly and ensuring that all participants can exchange opinions effectively.

[0582] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence and perform functions such as learning, reasoning, and recognition.

[0583] "Speech content" refers to opinions and information verbally expressed by participants in meetings and discussions.

[0584] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0585] "Vocal timbre" refers to the characteristics of a speaker's voice, such as timbre, tone, and rhythm, and is an element that conveys emotions and intentions.

[0586] "Quantifying emotions" means analyzing participants' emotional states and expressing them as numerical data.

[0587] A "factory meeting" is a meeting held at a manufacturing site, where discussions are held on product production, quality control, business improvements, and so on.

[0588] A "brainstorming session" is a discussion format in which participants freely exchange ideas and find creative solutions.

[0589] The "emotion engine" is a technology that analyzes data such as participants' facial expressions and tone of voice to detect their emotional state in real time.

[0590] "Real-time detection" means analyzing data instantly and obtaining results without delay.

[0591] "Coordinating the progress of a meeting" means managing the flow and progress of the meeting and intervening or correcting as necessary.

[0592] "Patent-specific facilitation" is a specialized support activity to promote patent-related discussions.

[0593] "Strategic-focused facilitation" is a specialized support activity aimed at promoting discussions on corporate and organizational strategies.

[0594] A system for implementing this invention includes means for automatically facilitating video conferences using AI, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating meetings, means for supporting meetings and brainstorming sessions within factories, and means for using an emotion engine to detect participants' emotions in real time and adjust the progress of the meeting.

[0595] The server first uses a camera (e.g., Logitech C920) to capture video of meeting participants in real time. Next, it uses an emotion detection engine called EmotionEngine to analyze the participants' emotions from the captured video and express them as numerical data. The analyzed emotional data is sent to a facilitation support AI called FacilitationAI, which suggests actions to optimize the progress of the meeting.

[0596] Specifically, if one participant shows anger or dissatisfaction, FacilitationAI instructs the server to provide that participant with an opportunity to speak. Furthermore, if a participant shows joy or satisfaction, it suggests actions to encourage further participation. This helps the meeting proceed smoothly and improves the satisfaction of all participants.

[0597] For example, if a meeting is being held at a factory to discuss the introduction of a new product line and one participant expresses dissatisfaction, the server can give that participant an opportunity to speak and convey their feelings to the other participants, thereby smoothing the meeting.

[0598] Examples of prompts for generative AI models include:

[0599] "We are having a meeting about introducing a new product line. One of the participants is expressing frustration. Please suggest what facilitation action should be taken in this situation."

[0600] In this way, the server can detect participants' emotions in real time and adjust the progress of the meeting based on that information, providing specialized facilitation for patents and strategies, and even in factory meetings and brainstorming sessions, it can properly manage participants' emotions and promote smooth communication.

[0601] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0602] Step 1:

[0603] The server captures video of the meeting participants in real time using a camera (e.g., Logitech C920). The input is the video data from the camera, and the output is the captured video frames, which are used in subsequent processing steps.

[0604] Step 2:

[0605] The server uses EmotionEngine to analyze participants' facial expressions from captured video frames and quantify their emotions. The input is the video frames, and the output is numerical data that indicates the emotional state of each participant. Specifically, EmotionEngine analyzes the video frames, detects facial feature points, and classifies emotions.

[0606] Step 3:

[0607] The server sends the analyzed emotional data to Facilitation AI, which then proposes actions to optimize the progress of the meeting. The input is emotional data, and the output is proposed facilitation actions. Specifically, Facilitation AI analyzes the emotional data and determines appropriate actions (e.g., providing an opportunity to speak, encouraging participation).

[0608] Step 4:

[0609] The server executes the actions proposed by FacilitationAI. The input is the proposed action, and the output is the result of the executed action. Specifically, the server coordinates the progress of the meeting, providing participants with opportunities to speak as needed and communicating their emotional state to other participants.

[0610] Step 5:

[0611] The server monitors the progress of the meeting and, if necessary, uses the Emotion Engine again to detect participants' emotions and send them to FacilitationAI. The input is the progress of the meeting and the latest video frames, and the output is updated emotional data and new facilitation actions. Specifically, the server periodically captures video frames and re-analyzes emotions to optimize the progress of the meeting.

[0612] Step 6:

[0613] After the meeting, the server saves all emotional data and the history of facilitation actions to help improve subsequent meetings. The input is the emotional data and the action history, and the output is the saved data. Specifically, the server saves the emotional data and the action history in a database and uses them as reference material for the next meeting.

[0614] Example 3

[0615] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0616] With conventional video conferencing systems, it was difficult to grasp the emotional state of attendees in real time and adjust the progress of the meeting appropriately. Furthermore, it was not possible to predict changes in attendees' emotions and optimize the progress of the meeting based on those predictions, which could result in a decline in the quality and effectiveness of the meeting. Furthermore, non-face-to-face communication often did not proceed smoothly, which increased the burden on users.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0618] In this invention, the server includes means for collecting attendee data, means for analyzing the collected data and evaluating the attendees' emotional states, means for adjusting the progress of the meeting based on the evaluation results, means for predicting attendees' emotional changes and optimizing the progress of the meeting based on the prediction, and means for providing smooth communication even when not face-to-face. This makes it possible to grasp the attendees' emotional states in real time and automatically provide appropriate facilitation. Furthermore, by predicting attendees' emotional changes and optimizing the progress of the meeting, the quality and effectiveness of the meeting can be improved. Furthermore, by providing smooth communication even when not face-to-face, the burden on users can be reduced.

[0619] "Means for collecting attendee data" refers to devices or software for collecting data such as facial expressions and voices of attendees in a meeting in real time.

[0620] The "means for analyzing the collected data and assessing the emotional state of attendees" refers to algorithms or software for analyzing the collected facial and voice data and assessing the emotional state of attendees (such as joy, anger, sadness, etc.).

[0621] The "means for adjusting the progress of the meeting based on the evaluation results" refers to a system or process for pausing the progress of the meeting or suggesting a break based on the evaluation results of the emotional state.

[0622] "Means for predicting changes in the emotions of attendees and optimizing the progress of the meeting based on the predictions" refers to algorithms or software that predict changes in the emotions of attendees based on past emotional data and adjust the progress of the meeting based on the prediction results.

[0623] "Means for providing smooth communication even when not face-to-face" refers to systems and processes that enable attendees to communicate smoothly with each other even when they are not physically face-to-face.

[0624] This invention is a system for facilitating non-face-to-face communication, and includes an AI function for automatically adjusting the progress of a meeting. Specific embodiments of this system are described below.

[0625] As soon as the meeting starts, the server collects facial expression and voice data from attendees via cameras and microphones connected to their devices (PCs or smartphones). The devices then send this data to the server in real time. The hardware used includes high-resolution cameras, microphones, and servers. The software used includes an emotion analysis engine (e.g., Microsoft Azure's Emotion API) and a meeting management system (e.g., Zoom or Microsoft Teams).

[0626] The server sends the collected facial and voice data to an emotion analysis engine, which analyzes the data and evaluates the attendees' emotional state (happiness, anger, sadness, surprise, etc.) in real time. For example, if an attendee's facial expression becomes grim and their voice tone becomes lower, the emotion analysis engine will determine that the attendee is expressing negative emotions. Based on this information, the server pauses the meeting and suggests that attendees take a break.

[0627] Furthermore, the emotion engine predicts changes in attendees' emotions based on past data. For example, if it predicts that a certain attendee is likely to be dissatisfied with the progress of the meeting, the server will provide that attendee with an opportunity to speak in advance, or take other action such as adjusting the progress of the meeting.

[0628] To illustrate, consider the following scenario:

[0629] Scenario: Participant A becomes frustrated during a meeting.

[0630] Processing: The emotion analysis engine analyzes Attendee A's facial expressions and tone of voice and detects dissatisfaction. The server pauses the meeting and gives Attendee A an opportunity to speak.

[0631] Example prompts to input to the generative AI model:

[0632] "Design a system that analyzes attendees' emotions in real time during a meeting, and if negative emotions are detected, pause the meeting and suggest a break. Also, add a function that predicts changes in attendees' emotions and optimizes the progress of the meeting."

[0633] In this way, the server analyzes the emotions of the attendees and automatically performs appropriate facilitation, thereby realizing an environment that provides smooth communication even without face-to-face communication. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0634] Step 1:

[0635] The server activates the camera and microphone connected to the device as soon as the meeting starts. The device captures the facial expressions and audio data of the participants in real time and sends them to the server. The input is the video and audio data from the camera and microphone, and the output is the raw data sent to the server.

[0636] Step 2:

[0637] The server sends the received facial expression and voice data to the emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the emotional state of the attendees. The input is the raw data sent from the server, and the output is the evaluation result of the emotional state (happiness, anger, sadness, surprise, etc.). Specifically, the emotion analysis engine analyzes facial expression features (e.g., eyebrow movement, mouth shape) and voice tone.

[0638] Step 3:

[0639] The server receives the evaluation results of the emotional state returned from the emotion analysis engine and determines whether the attendees' emotions are positive or negative. The input is the evaluation result from the emotion analysis engine, and the output is the judgment result of the emotional state. Specifically, if the server determines that the attendees' emotions are negative based on the evaluation results, it pauses the progress of the meeting.

[0640] Step 4:

[0641] If a negative emotion is detected, the server instructs the device to display a message suggesting a break. The input is the result of the emotional state judgment, and the output is the break suggestion message displayed on the device. Specifically, the server sends a message to the device such as, "The participant's emotions are leaning negative, so we suggest a 5-minute break."

[0642] Step 5:

[0643] The server predicts changes in the emotions of attendees based on past emotion data. The input is past emotion data, and the output is the predicted emotion change. Specifically, the emotion engine uses a machine learning algorithm to predict whether a particular attendee is likely to be dissatisfied with the progress of the meeting.

[0644] Step 6:

[0645] The server takes action to optimize the progress of the meeting based on the prediction of emotion changes. The input is the predicted emotion change result, and the output is instructions for the optimized progress of the meeting. Specifically, the server provides attendees who are predicted to be dissatisfied with the meeting with an opportunity to speak in advance. It also takes action such as adjusting the progress of the meeting.

[0646] In this way, the server analyzes the emotions of attendees and automatically provides appropriate facilitation, creating an environment that provides smooth communication even when not face-to-face.

[0647] (Application example 3)

[0648] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0649] With conventional video conferencing systems and customer service systems, it was difficult to properly analyze the emotions of attendees and customers and adjust responses in real time based on that analysis. As a result, meetings could not proceed smoothly, leading to lower participant satisfaction, and customer service could be inadequate, leading to lower customer satisfaction. There is a need to solve these issues and improve the quality of meetings and customer service.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0651] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for analyzing customers' facial expressions and tone of voice and determining their emotions in real time, means for proposing appropriate responses based on the results of the emotion analysis, and means installed on a smart device to support customer service. This makes it possible for the conference to proceed smoothly, improving the satisfaction of all participants, and for customer service to be provided quickly and appropriately, thereby improving customer satisfaction.

[0652] "Video conference facilitation" refers to the coordination and management of video conferences to ensure their smooth progress.

[0653] "Automated AI means" refers to methods or devices that allow artificial intelligence to automatically perform specific tasks.

[0654] "Means for analyzing the remarks, facial expressions, and tone of voice of attendees and quantifying their emotions" refers to a method or device for analyzing the remarks, facial expressions, and tone of voice of meeting participants and expressing their emotions as a number.

[0655] "Means for optimal meeting facilitation" refers to methods and devices for optimizing the progress of meetings.

[0656] "Means for analyzing a customer's facial expression and tone of voice to determine their emotions in real time" refers to a method or device for analyzing a customer's facial expression and tone of voice to determine their emotions in real time.

[0657] The term "means for proposing an appropriate response based on the result of emotion analysis" refers to a method or device for proposing an appropriate response based on the result of emotion analysis.

[0658] "Means installed on a smart device to assist in customer service" refers to a method or apparatus that is installed on a device such as a smartphone or smart glasses to assist in customer service.

[0659] This invention is a system that uses AI to analyze emotions during video conferences and customer service, and proposes appropriate responses in real time. This system has the functions of automatically facilitating video conferences to ensure smooth progress, and supporting customer service.

[0660] System configuration

[0661] The system consists of the following major components:

[0662] 1. Video conference facilitation module: AI automatically manages video conferences.

[0663] 2. Emotion analysis module: Analyzes the content, facial expressions, and tone of voice of attendees and customers, and quantifies their emotions.

[0664] 3. Response suggestion module: Based on the results of sentiment analysis, it suggests optimal meeting facilitation and customer response.

[0665] 4. Smart device interface: Installed on devices such as smartphones and smart glasses, it assists in customer service.

[0666] Hardware and software used

[0667] Hardware: smart glasses, smartphone, camera, microphone

[0668] Software: OpenCV (face detection), Keras (emotion analysis model)

[0669] Data processing and calculation

[0670] 1. Face detection: Capture the faces of customers and attendees in real time using smart glasses or smartphone cameras. Use OpenCV to detect faces and obtain facial expression data.

[0671] 2. Emotion analysis: The acquired facial expression data is input into the Keras emotion analysis model to quantify emotions, thereby determining the emotional state of customers and attendees in real time.

[0672] 3. Response suggestions: Based on the results of emotion analysis, appropriate responses are suggested. For example, if a customer is dissatisfied, the smart glasses display will say, "The customer is dissatisfied. Please improve your response."

[0673] Specific examples

[0674] scenario

[0675] If a store clerk is interacting with a customer and the customer looks dissatisfied, the smart glasses' display will display a message saying, "The customer is dissatisfied. Please improve your service." This allows the store clerk to quickly improve their service and increase customer satisfaction.

[0676] Prompt Sentence Examples

[0677] "Capture the customer's facial expressions and use an emotion analysis model to determine their emotions. If the customer is dissatisfied, display a message suggesting how to improve the experience."

[0678] In this way, a system that supports customer service in video conferences and brick-and-mortar stores can be realized. This system will enable meetings to proceed smoothly, improving the satisfaction of all participants, and enabling customers to be served promptly and appropriately, thereby improving customer satisfaction.

[0679] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0680] Step 1:

[0681] Face detection

[0682] The server captures the faces of customers and attendees in real time using smart glasses or a smartphone camera. As input, it receives video data from the camera and detects faces using OpenCV. As output, it obtains the position information and facial expression data of the detected faces. Specifically, it analyzes the camera video frame by frame and identifies the facial contours.

[0683] Step 2:

[0684] Emotion analysis

[0685] The server inputs the facial expression data obtained in step 1 into the Keras emotion analysis model. It receives facial position information and facial expression data as input and quantifies emotions using the emotion analysis model. The output is the emotional state of the customer or attendee (e.g., joy, sadness, anger, etc.). Specifically, it preprocesses the facial expression data and inputs it into the emotion analysis model to predict emotions.

[0686] Step 3:

[0687] Response proposal

[0688] The server proposes an appropriate response based on the emotion analysis results obtained in step 2. It receives the emotional state as input and determines the appropriate response using the response suggestion module. As output, it generates a message to be displayed on the display of the smart glasses or smartphone. Specifically, it searches a database for a response appropriate to the emotional state and generates a response message.

[0689] Step 4:

[0690] Message display

[0691] The device displays the corresponding message generated in step 3 on the display of the smart glasses or smartphone. As input, it receives the corresponding message and processes it to display it on the display. As output, it visually presents the message to the user. Specifically, it calls the display's API and displays the message.

[0692] Step 5:

[0693] User response

[0694] The user responds appropriately to the customer or attendees based on the message displayed in step 4. As input, the user receives the displayed message and implements a countermeasure. As output, the user improves the satisfaction of the customer or attendees. As a specific action, the user adjusts the customer response or the progress of the meeting according to the message.

[0695] In this way, the system can analyze emotions in real time and suggest appropriate responses, improving the quality of meetings and customer interactions.

[0696] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0697] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0698] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0699] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0700] [Second embodiment]

[0701] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0702] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0703] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0704] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0705] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0706] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0707] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0708] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0709] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0710] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0711] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0712] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0713] "Example 1"

[0714] One aspect of the present invention is a system that uses AI to automatically facilitate video conferences. Specifically, it is equipped with an AI engine that analyzes the content, facial expressions, and tone of voice of attendees. This AI engine uses voice recognition and image recognition technology to quantify attendees' emotions, and based on the results, optimally facilitates the conference.

[0715] "Example 2"

[0716] Furthermore, it also has the ability to provide specialized facilitation, such as patent-specific and strategy-specific facilitation. For example, in patent-specific facilitation, AI automatically asks questions and makes suggestions to promote patent-related discussions. This reduces the burden on users and improves the quality and effectiveness of meetings.

[0717] "Example 3"

[0718] It also has a function that provides an environment where smooth communication can be carried out even without face-to-face meetings. Specifically, the AI ​​analyzes the emotions of attendees and adjusts the progress of the meeting based on the results. For example, if it determines that attendees' emotions are leaning negatively, the AI ​​can automatically perform appropriate facilitation like a human would, such as pausing the meeting and suggesting that attendees take a break.

[0719] The processing flow of each embodiment will be described below.

[0720] "Example 1"

[0721] Step 1: The AI ​​engine starts the video conference and analyzes the participants' speech, facial expressions, and tone of voice in real time.

[0722] Step 2: The AI ​​engine uses voice and image recognition technology to quantify the emotions of attendees.

[0723] Step 3: The AI ​​engine uses the analysis results to optimally facilitate the meeting. For example, if it determines that attendees' emotions are leaning negatively, it will pause the meeting and suggest that attendees take a break.

[0724] "Example 2"

[0725] Step 1: The AI ​​engine provides specialized facilitation, such as patent-specific or strategy-specific facilitation.

[0726] Step 2: In patent-specific facilitation, AI automatically asks questions and makes suggestions to promote patent-related discussions.

[0727] Step 3: This reduces the burden on users and improves the quality and effectiveness of meetings.

[0728] "Example 3"

[0729] Step 1: Provide an environment where the AI ​​engine can communicate smoothly even without face-to-face contact.

[0730] Step 2: AI analyzes attendees' emotions and adjusts the meeting progress based on the results.

[0731] Step 3: For example, if the AI ​​determines that attendees' emotions are leaning negatively, it can automatically take appropriate facilitation steps, just like a human would, such as pausing the meeting and suggesting that attendees take a break.

[0732] Example 1

[0733] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0734] Conventional video conferencing systems have difficulty in properly analyzing the content, facial expressions, and tone of voice of attendees and optimizing the progress of the meeting. Furthermore, there was a lack of means to provide feedback to improve the quality and effectiveness of the meeting, which placed a heavy burden on users and hindered smooth non-face-to-face communication.

[0735] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0736] In this invention, the server includes a means for automatically facilitating the video conference using artificial intelligence, a means for analyzing the content of attendees' remarks, facial expressions, and tone of voice and quantifying their emotions, a means for converting voice data into text using speech recognition technology, a means for analyzing facial expressions from video data using image recognition technology, a means for optimal conference facilitation based on the quantified emotional data, and a means for providing feedback based on the progress of the conference and the emotional data. This makes it possible to analyze attendees' emotions in real time and ensure optimal conference progress. Furthermore, by providing feedback to improve the quality and effectiveness of the conference, the burden on users is reduced and smooth communication can be achieved even when not face-to-face.

[0737] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[0738] "Facilitation" is a technique for smoothly progressing meetings and group discussions and enabling participants to exchange opinions effectively.

[0739] "Artificial intelligence" refers to a computer system that mimics human intelligence and has capabilities such as learning, reasoning, and recognition.

[0740] "Speech" refers to the words and opinions expressed orally by participants during a meeting.

[0741] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0742] "Vocal timbre" refers to characteristics such as the quality of the voice, tone, and intonation, and is an element that conveys the speaker's emotions and intentions.

[0743] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[0744] "Quantifying emotions" means quantitatively evaluating emotions and expressing them as numbers.

[0745] "Speech recognition technology" is a technology that converts speech into text, and can automatically record what a speaker says as text.

[0746] "Image recognition technology" is a technology that analyzes image data and recognizes specific patterns and features.

[0747] "Optimal meeting facilitation" is a method for conducting a meeting most effectively, taking into account the circumstances and emotions of the participants.

[0748] "Feedback" refers to evaluations and advice provided based on the progress and results of the meeting.

[0749] "Progress" indicates how the meeting is progressing and its current status.

[0750] "Emotional data" is data that quantifies the emotions of participants and can be used to guide and facilitate meetings.

[0751] This invention relates to a system that uses artificial intelligence to automatically facilitate video conferences. This system analyzes the content of participants' remarks, facial expressions, and tone of voice, and quantifies their emotions to achieve optimal conference facilitation.

[0752] Hardware and software used

[0753] Hardware: Camera, microphone, device (PC, tablet, smartphone)

[0754] Software: speech recognition technology (e.g., Google Speech-to-Text), image recognition technology (e.g., OpenCV), AI engine (e.g., TensorFlow)

[0755] System Operation

[0756] Subject: Server

[0757] The server provides a system that automatically facilitates video conferences. This system is equipped with an AI engine that analyzes the content of participants' remarks, facial expressions, and tone of voice. The server uses voice recognition and image recognition technology to quantify the emotions of participants. Based on this quantified emotional data, the server provides optimal conference facilitation. It also provides feedback based on the progress of the conference and the emotional data.

[0758] Subject: Terminal

[0759] The device transmits video and audio from the video conference to the server. The device also uses a camera and microphone to capture the facial expressions and speech of participants. This data is sent to the server in real time and analyzed by an AI engine.

[0760] Subject: User

[0761] Users participate in video conferences and send their comments and facial expressions to the server via their devices. Users simply participate in the conference in a natural way, without any special operations. The AI ​​engine analyzes the content of the user's comments, facial expressions, and tone of voice, providing optimal conference facilitation.

[0762] Specific examples

[0763] Example 1: Analysis of speech during a meeting

[0764] When User A says, "How is the progress on this project?", the server uses voice recognition technology to convert the speech into text. The AI ​​engine then analyzes this text and recognizes that User A is asking a question. The server then facilitates the meeting by encouraging other attendees to provide appropriate answers.

[0765] Example 2: Facial expression analysis

[0766] The device's camera captures User B's facial expression and sends it to the server. The server uses image recognition technology to analyze User B's facial expression and detects that User B is confused. The server pauses the meeting and facilitates by asking User B questions or opinions.

[0767] Prompt Sentence Examples

[0768] "Please describe a system that analyzes what participants say, their facial expressions, and tone of voice during a video conference to provide optimal meeting facilitation. Please include the names of any specific hardware or software."

[0769] In this way, the system's operation can be explained from the perspectives of the server, the device, and the user, and concrete examples are used to deepen understanding. This system can analyze attendees' emotions in real time and ensure optimal meeting progress. It also provides feedback to improve the quality and effectiveness of meetings, reducing the burden on users and enabling smooth communication even when not face-to-face.

[0770] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0771] Step 1: Start a video conference

[0772] Subject: User

[0773] A user starts a video conference using a terminal. The user starts a conference application and enters the conference ID to join. For example, when a user joins a conference using a conference application, the user enters the conference ID and clicks the "Join" button.

[0774] Input: Meeting ID, user operation

[0775] Output: Start a video conference and prepare to capture video and audio

[0776] Step 2: Capturing Data

[0777] Subject: Terminal

[0778] The device uses a camera and microphone to capture the user's video and audio. The camera captures the user's facial expressions in real time, and the microphone records what the user says. For example, the device's camera captures the user's smile, and the microphone records when the user says "hello."

[0779] Input: User video and audio

[0780] Output: Captured video and audio data

[0781] Step 3: Sending data

[0782] Subject: Terminal

[0783] The device sends the captured video and audio data to the server. The data is encrypted and sent securely to the server. For example, the device sends the captured video and audio data to the server using the HTTPS protocol.

[0784] Input: Captured video and audio data

[0785] Output: Data sent to the server

[0786] Step 4: Analyze the data

[0787] Subject: Server

[0788] The server analyzes the received video and audio data. It uses voice recognition technology to convert the audio data into text and image recognition technology to analyze facial expressions from the video data. For example, the server uses voice recognition technology to convert the audio of "hello" into text and image recognition technology to detect the user's smile.

[0789] Input: Video and audio data sent to the server

[0790] Output: Text data, facial expression analysis results

[0791] Step 5: Quantify emotions

[0792] Subject: Server

[0793] The server quantifies the user's emotions based on the analysis results. The AI ​​engine analyzes changes in voice tone and facial expressions to generate an emotion score. For example, the server quantifies the user's smile as "joy" and assigns an emotion score of 80 / 100.

[0794] Input: Text data, facial expression analysis results

[0795] Output: Sentiment score

[0796] Step 6: Facilitation

[0797] Subject: Server

[0798] The server then optimally facilitates the meeting based on the quantified emotional data. For example, if a user is confused, the server may pause the meeting and prompt the user to ask questions or give their opinions.

[0799] Input: Sentiment score

[0800] Output: Facilitation instructions

[0801] Step 7: Provide feedback

[0802] Subject: Server

[0803] The server provides feedback based on the progress of the meeting and emotional data. For example, after the meeting, it sends a report to attendees about changes in their emotions and the frequency of their comments during the meeting.

[0804] Input: Meeting progress, emotion data

[0805] Output: Feedback report

[0806] In this way, by explaining each processing step specifically and clearly showing the inputs and outputs, the operation of the system can be more clearly understood.

[0807] (Application example 1)

[0808] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0809] With conventional video conferencing systems, it was difficult to analyze participants' remarks, facial expressions, and tone of voice to quantify their emotions and provide optimal meeting facilitation. Furthermore, while efficient progress is required for factory meetings and discussions, there was a lack of automated means for conducting meetings that took participants' emotions into consideration. This resulted in a decline in the quality and effectiveness of meetings and an increased burden on users.

[0810] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0811] In this invention, the server includes means for automatically faciliting video conferences using AI, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating the conference, means to be installed in a robot for streamlining in-plant meetings and discussions, and means for optimizing the progress of the conference taking into account the emotions of attendees.This makes it possible to efficiently conduct in-plant meetings and discussions while taking into account the emotions of attendees.

[0812] A "video conference" is a conference in which multiple participants share video and audio in real time via the Internet.

[0813] "Facilitation" refers to methods and techniques for smoothly conducting meetings and discussions.

[0814] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to learn and reason by imitating human intelligence.

[0815] "Speech content" refers to the information and opinions that participants verbally express during a meeting or discussion.

[0816] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0817] "Vocal timbre" refers to the tone and texture of a voice, and is an element that reflects the speaker's emotions and intentions.

[0818] "Quantifying emotions" means analyzing participants' emotional states and expressing them as numerical data.

[0819] "Optimal meeting facilitation" means taking into consideration the content and emotional state of participants and conducting the meeting in the most effective manner.

[0820] A "factory meeting" is a meeting held to discuss matters related to factory operations and production.

[0821] A "meeting" is when multiple people gather to discuss a specific purpose or issue and exchange opinions and share information.

[0822] A "robot" is a mechanical device that operates automatically according to a program to perform specific tasks.

[0823] "Installation" means introducing software or applications into a computer or machine device and making them ready for use.

[0824] "Optimizing the progress of a meeting" means making the necessary adjustments and improvements to ensure that the meeting proceeds efficiently and effectively.

[0825] As an embodiment of this invention, we will explain a "smart conference robot" system for streamlining meetings and discussions within factories. This system uses AI to automatically facilitate video conferences, analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions and provide optimal conference facilitation.

[0826] System configuration

[0827] 1. Hardware:

[0828] Camera: Used to capture attendees' facial expressions in real time.

[0829] Microphone: Used to capture what attendees are saying.

[0830] Robot: A mechanical device that moves around the factory and helps run meetings.

[0831] 2. Software:

[0832] Speech Recognition Library: Use the speech_recognition library to recognize what attendees are saying in real time.

[0833] Emotion analysis model: Using the transformers library pipeline, we analyze attendees' facial expressions and tone of voice to quantify their emotions.

[0834] Image processing library: Uses the cv2 library to acquire images from the camera in real time and perform facial expression analysis.

[0835] System Operation

[0836] 1. Speech Recognition:

[0837] The server converts the audio data acquired from the microphone into text data using the speech_recognition library.

[0838] The converted text data is used as information for optimizing the progress of the conference.

[0839] 2. Emotion analysis:

[0840] The server processes the video data acquired from the camera in real time using the cv2 library and analyzes the facial expressions of the attendees.

[0841] The analyzed facial expression data is used to quantify emotions using a pipeline from the transformers library.

[0842] 3. Optimizing meeting progress:

[0843] The server executes logic to optimize the progress of the conference based on the acquired voice data and emotion data.

[0844] For example, if an attendee says, "There's a problem with this part," the robot will respond, "There's a problem. We'll suggest a solution."

[0845] Also, if the attendee's facial expression is negative, the message "The attendee's emotions are negative. We will adjust the meeting progress." will be displayed.

[0846] Specific examples

[0847] As a concrete example, consider a quality control meeting in a factory. If a participant says, "There is a problem with this part," the robot responds, "A problem has occurred. We will propose a solution." If the participant's facial expression is negative, the robot displays, "The participant's emotions are negative. We will adjust the progress of the meeting."

[0848] Prompt Sentence Examples

[0849] Examples of prompts to input to a generative AI model include:

[0850] If someone says, "We have a problem with this part," please suggest how the meeting should proceed.

[0851] In this way, the smart conference robot system can make meetings within factories more efficient and provide optimal facilitation that takes into account the emotions of attendees.

[0852] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0853] Step 1:

[0854] The server acquires video data from the camera in real time. The input is video data from the camera, and the output is frame data processed using the image processing library cv2. Specifically, the server captures the video from the camera frame by frame, converts it to grayscale, and performs face detection.

[0855] Step 2:

[0856] The server acquires voice data from the microphone in real time. The input is the voice data from the microphone, and the output is the speech content converted into text data using the speech recognition library, speech_recognition. Specifically, the server records the voice from the microphone and converts it into text using a speech recognition engine.

[0857] Step 3:

[0858] The server analyzes the facial expressions of attendees using the acquired frame data. The input is the frame data obtained in step 1, and the output is emotional data quantified using the emotion analysis model transformers pipeline. Specifically, the server extracts the area where the face was detected, inputs it into the emotion analysis model, and quantifies the emotion.

[0859] Step 4:

[0860] The server analyzes the acquired text data and extracts information necessary for the progress of the meeting. The input is the text data obtained in step 2, and the output is instructions and suggestions for the progress of the meeting. Specifically, the server inputs the text data into a natural language processing engine and extracts important keywords and phrases.

[0861] Step 5:

[0862] The server executes logic to optimize the progress of the meeting based on the emotion data and text data. The input is the emotion data obtained in step 3 and the text data obtained in step 4, and the output is specific actions for the progress of the meeting. As a specific operation, the server integrates the emotion data and text data and generates instructions for adjusting the progress of the meeting.

[0863] Step 6:

[0864] The server sends the generated instructions to the robot to support the progress of the meeting. The input is the instructions for proceeding with the meeting obtained in step 5, and the output is specific actions by the robot. As specific actions, the server sends voice and action instructions to the robot to support the progress of the meeting.

[0865] Step 7:

[0866] The user follows instructions from the robot to conduct the meeting. The input is instructions from the robot, and the output is the progress of the meeting. In concrete terms, the user speaks and acts according to the robot's instructions to ensure the meeting proceeds smoothly.

[0867] Example 2

[0868] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0869] In conventional video conferencing systems, meeting facilitation is done manually, which places a heavy burden on attendees and can reduce the quality and effectiveness of meetings. Furthermore, patent-related discussions require specialized knowledge, making it difficult to generate appropriate questions and proposals. Furthermore, non-face-to-face communication often does not proceed smoothly, causing meetings to stall.

[0870] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0871] In this invention, the server includes a means for automatically facilitating video conferences using artificial intelligence, a means for analyzing participants' remarks, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for collecting patent-related data, a means for preprocessing the collected data, a means for generating questions and suggestions using a generative AI model, a means for transmitting the generated questions and suggestions to a terminal, and a means for the terminal to present the results to the user. This reduces the burden on users and improves the quality and effectiveness of conferences. It also promotes specialized discussions related to patents and enables smooth communication even when not face-to-face.

[0872] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[0873] "Facilitation" is the activity of providing support and coordination to ensure the smooth progress of meetings and discussions.

[0874] "Artificial intelligence" is the technology that enables computer systems to learn, reason, and self-correct by imitating human intelligence.

[0875] "Speech content" refers to information and opinions verbally expressed by participants during a meeting or discussion.

[0876] "Facial expressions" are emotions and reactions shown by the movement of facial muscles.

[0877] "Vocal timbre" refers to the characteristics of a voice, such as its quality, tone, and pitch.

[0878] "Quantifying emotions" means expressing emotional states as numerical data.

[0879] A "patent" is a legal right that grants an inventor exclusive use of an invention for a certain period of time.

[0880] "Collecting data" is the act of gathering the necessary information.

[0881] "Preprocessing" is the initial process used to prepare data for easy analysis and use.

[0882] A "generative AI model" is a model that uses artificial intelligence to generate new data and information.

[0883] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific topic.

[0884] A "terminal" is a device that allows a user to input and output information.

[0885] A "user" is someone who uses a system or service.

[0886] "Presenting the results" means showing the generated information or data to the user.

[0887] This invention is a system that uses artificial intelligence to automatically facilitate video conferences. The system has the ability to analyze participants' remarks, facial expressions, and tone of voice to quantify their emotions. It also includes the ability to collect patent-related data, perform preprocessing, and generate questions and suggestions using a generative AI model. A specific embodiment of this system is described below.

[0888] Hardware and Software Configuration

[0889] server

[0890] The server comprises hardware and software to perform the following functions:

[0891] Data collection: The server collects the necessary information from patent-related databases (e.g., patent information databases).

[0892] Data preprocessing: The server uses Python's NLTK library and spaCy to preprocess the collected data.

[0893] Applying generative AI models: The server uses a pre-trained generative AI model (e.g., GPT-4) to generate patent-related questions and suggestions.

[0894] Sending results: The server sends the generated questions and suggestions to the device.

[0895] Terminal

[0896] The terminal is a device where the user can input information and display the results sent from the server. The terminal has the following functions:

[0897] Receiving user input: The terminal receives prompts entered by the user.

[0898] Displaying results: The device presents the user with the questions and suggestions sent by the server.

[0899] User

[0900] A user is someone who uses the system to hold a video conference. The user performs the following operations:

[0901] Entering a prompt: The user enters a prompt into the terminal to facilitate discussion related to the patent.

[0902] Specific examples

[0903] For example, consider a case where a user uses this system when holding a meeting regarding a patent.

[0904] Example prompt sentence:

[0905] "I'd like to discuss a new patent application. Please generate questions and suggestions regarding the technical details of the patent and the competitive situation in the market."

[0906] When a user enters this prompt into a terminal, the server collects and preprocesses relevant data from a patent information database. It then uses a generative AI model to generate appropriate questions and suggestions, which it then sends to the terminal. The terminal then presents the generated questions and suggestions to the user. For example, a question such as "How is this technology different from existing patents?" is displayed.

[0907] In this way, users can use the system to efficiently advance patent-related discussions. The system reduces the burden on users and improves the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[0908] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0909] Step 1:

[0910] The user enters a prompt statement.

[0911] The user inputs a prompt sentence into the terminal to promote discussions related to patents. For example, the user inputs, "I would like to hold a discussion about a new patent application. Please generate questions or suggestions about the technical details of the patent and the competitive situation in the market." The input prompt sentence is sent from the terminal to the server.

[0912] Step 2:

[0913] The server collects the data.

[0914] The server receives the prompt sent by the user and collects data related to patents. Specifically, it retrieves relevant patent information from a patent information database. For example, it calls the API of a patent information database to collect patent information related to "new technology." The input is the prompt, and the output is the collected patent information.

[0915] Step 3:

[0916] The server preprocesses the data.

[0917] The server preprocesses the collected patent information. It uses Python's NLTK library and spaCy to analyze the text data and extract important keywords and phrases. For example, it extracts keywords such as "technical details" and "market competitive situation." The input is the collected patent information, and the output is the preprocessed data.

[0918] Step 4:

[0919] The server applies the generative AI model.

[0920] The server generates questions and suggestions using a generative AI model (e.g., GPT-4) based on the preprocessed data. The generative AI model generates appropriate questions and suggestions based on the extracted keywords. For example, it generates questions such as, "How is this technology different from existing patents?" The input is the preprocessed data, and the output is the generated questions and suggestions.

[0921] Step 5:

[0922] The server transmits the generated results to the terminal.

[0923] The server sends the generated questions and suggestions to the terminal. The generated questions and suggestions are sent to the terminal in JSON format. For example, they are sent in the format {"question": "How is this technology different from existing patents?"}. The input is the generated questions and suggestions, and the output is the data sent to the terminal.

[0924] Step 6:

[0925] The terminal presents the results to the user.

[0926] The terminal presents the questions and suggestions received from the server to the user. The user can then advance the discussion based on the presented questions and suggestions. For example, the terminal screen might display "How is this technology different from existing patents?" The input is the data sent from the server, and the output is the questions and suggestions presented to the user.

[0927] (Application example 2)

[0928] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0929] In conventional video conferencing systems, meeting facilitation is done manually, which places a heavy burden on attendees and can reduce the quality and effectiveness of meetings. Furthermore, patent-related discussions and technical debates require specialized knowledge, making it difficult to ask appropriate questions or make appropriate suggestions. Furthermore, the inability to smoothly communicate without face-to-face contact is also an issue.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0931] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing attendees' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for generating questions and suggestions to promote patent-related discussions, and means for generating prompt sentences based on the conference topic using a generative AI model.This improves the quality and effectiveness of conferences, efficiently supports patent-related discussions and technical debates, and enables smooth communication even when not face-to-face.

[0932] A "video conference" is a conference in which multiple participants share video and audio in real time via the Internet.

[0933] "Facilitation" is the activity of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[0934] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0935] "Speech content" refers to information or opinions expressed orally by participants in a meeting or discussion.

[0936] "Facial expressions" refer to emotions and reactions expressed through the movement of facial muscles.

[0937] "Vocal timbre" refers to characteristics such as the quality of a voice, tone, and intonation.

[0938] "Quantifying emotions" means expressing emotional states as numerical data.

[0939] A "patent" is an exclusive right granted to an invention.

[0940] "Facilitating discussion" means supporting participants to actively exchange opinions and engage in lively debate.

[0941] "Generating questions and suggestions" refers to automatically creating appropriate questions and suggestions based on a specific topic.

[0942] A "generative AI model" is an artificial intelligence model that generates new data or information based on input data.

[0943] A "prompt" is an instruction or question that is input into a generative AI model.

[0944] "Inside a factory" refers to the inside of a facility where production activities in the manufacturing industry take place.

[0945] A "technical discussion" is an exchange of opinions and discussion between experts on technical matters.

[0946] A system for carrying out the present invention includes a program for automatically facilitating a video conference. The system is realized using the following hardware and software.

[0947] Hardware

[0948] Server: A server with high-performance computing power is required to run generative AI models and analyze data quickly.

[0949] Device: You will need a device (computer, tablet, smartphone, etc.) to participate in the video conference.

[0950] Camera and microphone: A camera and microphone are required to capture what participants are saying, their facial expressions, and tone of voice.

[0951] software

[0952] Generative AI model: Uses OpenAI's API to generate prompts based on the meeting topic.

[0953] Data analysis software: Software is needed to analyze what attendees say, their facial expressions, and tone of voice, and to quantify their emotions.

[0954] Video conferencing software: Software for conducting video conferences (e.g., Zoom or Microsoft Teams) is required.

[0955] Data processing and calculation

[0956] The server analyzes participants' comments, facial expressions, and tone of voice in real time during video conferences to quantify their emotions. This allows the system to grasp the progress of the conference and participants' reactions. Furthermore, it uses a generative AI model to generate prompts based on the conference topic, automatically asking questions and making suggestions to promote patent-related discussions.

[0957] Specific examples

[0958] For example, if a meeting is being held regarding the patenting of a new manufacturing process, the server might generate a prompt like this:

[0959] "What technical features should be highlighted to patent this manufacturing process?"

[0960] "What are the advantages of this process compared to competitor patents?"

[0961] This allows attendees to efficiently advance discussions, improving the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[0962] In this way, the system for implementing the invention can automate the facilitation of video conferences to support patent-related discussions and technical debates.

[0963] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0964] Step 1:

[0965] At the start of a video conference, the server activates the camera and microphone to capture what participants are saying, their facial expressions, and tone of voice. As input, it receives real-time data from the camera and microphone, and as output, it sends this data to software for analysis.

[0966] Step 2:

[0967] The server uses analysis software to analyze the acquired data on speech content, facial expressions, and tone of voice, and quantifies emotions. It receives real-time data from the camera and microphone as input, and generates quantified emotional data as output. Specifically, it converts speech content into text using voice recognition technology, and analyzes facial muscle movements using facial expression recognition technology.

[0968] Step 3:

[0969] Based on the analysis results, the server uses a generative AI model to generate a prompt based on the meeting topic. It receives quantified emotion data and the meeting topic as input, and generates a prompt as output. Specifically, it generates a prompt using OpenAI's API.

[0970] Step 4:

[0971] The server sends the generated prompt sentence to the video conferencing software and presents it to the attendees. The server receives the generated prompt sentence as input and displays the prompt sentence on the video conferencing software as output. As a specific operation, the server displays the prompt sentence using a chat function or a screen sharing function of the video conferencing software.

[0972] Step 5:

[0973] The user advances the discussion based on the presented prompt sentence. The system receives the prompt sentence as input and generates the content of the discussion as output. As a specific action, the user expresses their opinion or question in response to the prompt sentence.

[0974] Step 6:

[0975] The server monitors the progress of the discussion and generates additional prompts as needed. It receives the content and emotion data of the discussion as input and generates additional prompts as output. Specifically, it uses the generative AI model again to generate new prompts.

[0976] Step 7:

[0977] At the end of the meeting, the server saves all data and generates a report of the meeting. It receives all data acquired during the meeting as input and generates a report of the meeting as output. Specific operations include saving the data in a database and creating a report using report generation software.

[0978] Example 3

[0979] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0980] With conventional video conferencing systems, it was difficult to analyze attendees' emotions in real time and adjust the progress of the meeting appropriately. This resulted in problems such as a decline in the quality and effectiveness of the meeting and an increase in the burden on users. In addition, non-face-to-face communication was often not possible, which caused the progress of the meeting to be delayed.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for automatically faciliting the conference using artificial intelligence, means for analyzing the content of attendees' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for collecting audio data of the conference, means for transmitting the collected audio data to the server, means for the server to perform emotion analysis of the audio data, means for adjusting the progress of the conference based on the analysis results, and means for suggesting appropriate actions to the user. This improves the quality and effectiveness of the conference, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0982] "Means for automated meeting facilitation using artificial intelligence" refers to a function that uses artificial intelligence to automatically progress and coordinate meetings.

[0983] "Means of analyzing attendees' comments, facial expressions, and tone of voice and quantifying their emotions" is a function that analyzes attendees' comments, facial expressions, and tone of voice and expresses their emotions as a number.

[0984] "Means for optimal meeting facilitation" is a function that makes adjustments to optimize the progress of meetings based on analyzed emotional data.

[0985] The "means for collecting conference audio data" is a function for collecting audio during a conference in real time.

[0986] The "means for transmitting collected voice data to a server" is a function for transmitting collected voice data to a server via the Internet.

[0987] The "means for the server to analyze the emotions of the voice data" is a function for the server to analyze the voice data received and identify the emotions of the attendees.

[0988] "Means for adjusting the progress of the meeting based on the analysis results" is a function for appropriately adjusting the progress of the meeting based on the results of emotion analysis.

[0989] The "means for suggesting appropriate actions to the user" is a function for suggesting appropriate actions to the user based on the analysis results.

[0990] This invention relates to a system that uses artificial intelligence to automatically facilitate meetings. This system analyzes the content, facial expressions, and tone of voice of attendees and quantifies their emotions to enable optimal meeting facilitation.

[0991] 1. Generate the system program

[0992] The system was developed using Python, with the main libraries being "Hugging Face Transformers" for emotion analysis and "Google Calendar API" for meeting management.

[0993] 2. Explain the program's processing in natural language

[0994] When a user starts a meeting, the device collects audio data in real time. The collected audio data is sent to a server, where it undergoes emotion analysis using Hugging Face Transformers. The analysis results are classified into three categories: positive, negative, and neutral.

[0995] The server adjusts the progress of the meeting based on the analysis results. For example, if it determines that attendees' emotions are leaning negatively, the server will pause the meeting through the Google Calendar API and display a message on the device suggesting a break.

[0996] 3. Add examples to your description

[0997] As a concrete example, the following scenario can be considered.

[0998] scenario:

[0999] A user starts an online meeting. During the meeting, one of the attendees repeatedly makes negative comments. The system performs sentiment analysis on the comments and determines that the comments have a strong negative sentiment. The server pauses the meeting and displays a message on the device asking, "Suggest taking a break?"

[1000] Example prompt sentence:

[1001] "Develop a system that analyzes attendees' emotions in real time during online meetings and suggests pausing the meeting and taking a break if negative emotions are present."

[1002] This system provides an environment in which smooth communication can be achieved even without face-to-face contact. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1003] Step 1:

[1004] A user starts a conference. The user launches the conference application and clicks the "Start Conference" button. This causes the system to start collecting voice data. The input is the user's operation, and the output is the conference start signal.

[1005] Step 2:

[1006] The device collects the audio data of the meeting. The device picks up the audio during the meeting with a microphone and converts it into digital audio data in real time. For example, when a user says "Hello, everyone," the audio is immediately collected as digital data. The input is the audio during the meeting, and the output is digital audio data.

[1007] Step 3:

[1008] The terminal sends the collected voice data to the server. The terminal sends the collected voice data to the server using the HTTPS protocol. The data is AES encrypted and transferred securely. The input is digital voice data, and the output is a transmission completion signal to the server.

[1009] Step 4:

[1010] The server performs emotion analysis on the voice data. The server inputs the received voice data into the Hugging Face Transformers emotion analysis model. For example, a statement such as "This project might not work out" is analyzed as negative. The input is digital voice data, and the output is the emotion analysis result.

[1011] Step 5:

[1012] The server adjusts the progress of the meeting based on the analysis results. If the server determines that the result of the sentiment analysis is negative, it issues an instruction to pause the progress of the meeting using the Google Calendar API. The input is the sentiment analysis result, and the output is an instruction to adjust the progress of the meeting.

[1013] Step 6:

[1014] The terminal suggests appropriate actions to the user. The terminal receives instructions from the server and displays a message to the user saying, "Attendees' emotions are negative. Would you like to suggest taking a break?" The user adjusts the progress of the meeting by selecting "Yes" or "No." The input is an instruction from the server, and the output is a suggested message to the user.

[1015] In this way, the system provides an environment in which smooth communication can be carried out even without face-to-face contact.

[1016] (Application example 3)

[1017] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1018] In conventional video conferencing systems and work environments, it was difficult to properly analyze the emotions of attendees and workers and optimize meeting progress and work efficiency. Furthermore, when emotions turned negative, it was not possible to respond appropriately, which led to problems with reduced meeting quality and work safety. This increased the burden on users and hindered smooth non-face-to-face communication.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1020] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time to improve work efficiency and safety, and means for pausing work and suggesting a break when emotions become negative.This optimizes the progress of the conference and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1021] A "video conference" is a conference in which multiple participants share video and audio via the Internet.

[1022] "Facilitation" means providing support and coordination to ensure that meetings and work progress smoothly.

[1023] "AI" stands for artificial intelligence, a technology that allows machines to imitate human intelligence and learn and reason.

[1024] "Speech" refers to information and opinions that participants verbally express during a meeting or task.

[1025] "Facial expression" refers to emotions and intentions shown by facial muscle movements.

[1026] "Tone" refers to the quality and tone of a voice, and reflects the speaker's emotions and state.

[1027] "Quantifying emotions" means expressing emotions as quantitative data.

[1028] A "worker" is someone who performs a specific task in a factory, office, etc.

[1029] "Real time" means dealing with what's happening right now.

[1030] "Work efficiency" refers to the ability to minimize waste of time and effort while performing work and achieve maximum results.

[1031] "Safety" means a state in which work or activities can be carried out without risk and with peace of mind.

[1032] "Suggesting a break" means temporarily pausing work or the progress of a meeting to encourage participants to take a rest.

[1033] A "system" is a mechanism in which multiple elements work together to perform a specific function.

[1034] A system for implementing this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating meetings, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for pausing work and suggesting a break when emotions become negative.

[1035] System Program

[1036] The program of this system performs the following processing.

[1037] 1. Hardware: The system uses a camera (e.g., Logitech C920) to capture the faces of attendees and workers.

[1038] 2. Software: The system uses software such as OpenCV, Keras, and TensorFlow to process the captured images and analyze emotions.

[1039] 3. Data processing: The captured image is converted to grayscale and face detection is performed. The detected face regions are preprocessed to be input into the emotion analysis model.

[1040] 4. Data computation: Predict emotions from preprocessed facial images using an emotion analysis model.

[1041] Specific examples of processing

[1042] For example, if a worker in a factory feels stressed, the system will analyze their emotions in real time and suggest pausing work and taking a break, improving work efficiency and safety while reducing the burden on the worker.

[1043] Prompt Sentence Examples

[1044] An example of a prompt to input to a generative AI model is as follows:

[1045] Develop a robot assistant application in a factory that analyzes workers' emotions in real time to improve work efficiency and safety. Include a function that allows the robot to pause work and suggest a break if the worker is feeling stressed.

[1046] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1047] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1048] Step 1:

[1049] The server uses a camera to capture the faces of attendees and workers. The input is video data from the camera, and the output is the captured image data. Specifically, the camera acquires video in real time and captures the video frame by frame.

[1050] Step 2:

[1051] The server converts the captured image data to grayscale. The input is the captured image data, and the output is grayscale image data. Specifically, it uses OpenCV to convert the image to grayscale.

[1052] Step 3:

[1053] The server detects faces from grayscale images. The input is grayscale image data, and the output is coordinate data of the detected face area. Specifically, the server identifies the face area using the OpenCV face detection algorithm.

[1054] Step 4:

[1055] The server preprocesses the detected face region to input it into the emotion analysis model. The input is the coordinate data and grayscale image data of the face region, and the output is the preprocessed face image data. Specifically, the server cuts out the face region, resizes it to 48x48 pixels, and normalizes it.

[1056] Step 5:

[1057] The server inputs the preprocessed facial image data into the emotion analysis model to predict emotions. The input is the preprocessed facial image data, and the output is the emotion prediction result. Specifically, Keras is used to input data into the emotion analysis model and obtain the prediction result.

[1058] Step 6:

[1059] The server determines whether the predicted emotion is negative. The input is the emotion prediction result, and the output is the negative emotion judgment result. Specifically, the server analyzes the prediction result and checks whether it contains negative emotions (e.g., anger, sadness, fear).

[1060] Step 7:

[1061] If a negative emotion is detected, the server pauses the work and suggests taking a break. The input is the negative emotion judgment result, and the output is a notification suggesting a break. Specifically, the system pauses the work and sends a notification to the user to take a break.

[1062] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1063] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1064] "Example 1"

[1065] In one embodiment of the present invention, the emotion engine recognizes a user's emotions and quantifies them. Specifically, the emotion is estimated from the user's remarks, tone of voice, facial expressions, etc., and quantified. This quantified emotion data is used for meeting facilitation. For example, if the AI ​​determines that the emotions of all participants are leaning negatively, it will pause the meeting and take action such as suggesting a break.

[1066] "Example 2"

[1067] The emotion engine also detects changes in the user's emotions in real time and adjusts the meeting facilitation accordingly. For example, if it senses that a participant is showing anger or dissatisfaction, the AI ​​will take action such as offering that participant an opportunity to speak or conveying that emotion to other participants. This helps the meeting proceed smoothly and improves the satisfaction of all participants.

[1068] "Example 3"

[1069] Furthermore, the emotion engine predicts changes in the user's emotions and optimizes meeting facilitation based on the predictions. For example, if it predicts that a certain participant is likely to be dissatisfied with the progress of the meeting, the AI ​​will take action such as providing that participant with an opportunity to speak in advance or adjusting the progress of the meeting. This will help the meeting proceed more smoothly and further improve the satisfaction of all participants.

[1070] The processing flow of each embodiment will be described below.

[1071] "Example 1"

[1072] Step 1: Estimate the user's emotions from their speech, tone of voice, facial expressions, etc.

[1073] Step 2: Quantify the estimated emotions.

[1074] Step 3: Use quantified emotional data to facilitate meetings.

[1075] Step 4: If the overall sentiment of the participants is judged to be negative, the AI ​​will take action such as pausing the meeting and suggesting a break.

[1076] "Example 2"

[1077] Step 1: The emotion engine detects changes in the user's emotions in real time.

[1078] Step 2: Adjust your meeting facilitation as emotions change.

[1079] Step 3: If the AI ​​senses that a participant is showing anger or dissatisfaction, it takes action such as offering that participant an opportunity to speak or conveying their feelings to other participants.

[1080] "Example 3"

[1081] Step 1: The emotion engine predicts changes in the user's emotions.

[1082] Step 2: Optimize meeting facilitation based on predictions.

[1083] Step 3: If it predicts that a participant is likely to be dissatisfied with the progress of the meeting, the AI ​​will take action such as providing that participant with an opportunity to speak in advance or adjusting the progress of the meeting.

[1084] Example 1

[1085] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1086] With conventional video conferencing systems, it was difficult to analyze attendees' emotions and comments in real time and optimize the progress of the meeting. There was also a lack of means to provide professional facilitation to improve the quality and effectiveness of meetings. Furthermore, there was no environment in place to reduce the burden on users and ensure smooth communication in non-face-to-face communication.

[1087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1088] In this invention, the server includes means for capturing users' speech content and facial expressions in real time using a high-resolution camera and microphone, means for transmitting the captured audio data and video data to the server, means for converting the audio data into text using voice recognition software, means for analyzing the user's facial expressions from the video data using image recognition software, means for quantifying the user's emotions from the text data and facial expression data using an emotion analysis engine, and means for optimizing the progress of the conference based on the quantified emotional data. This makes it possible to analyze attendees' emotions and speech content in real time and provide optimal conference facilitation.

[1089] A "high-resolution camera" is a camera capable of capturing high-resolution images to capture the user's face and facial expressions in detail.

[1090] A "microphone" is a voice input device for accurately recording what a user says.

[1091] "Voice data" refers to voice information including the content of a user's speech acquired through a microphone.

[1092] "Video data" refers to video information including the user's face and facial expressions captured through a high-resolution camera.

[1093] A "server" is a computer system for receiving and analyzing audio and video data.

[1094] "Speech recognition software" is software that analyzes voice data and converts it into corresponding text data.

[1095] "Text data" is text information generated from voice data by voice recognition software.

[1096] "Image recognition software" is software that analyzes video data and recognizes the user's facial expressions.

[1097] An "emotion analysis engine" is an analysis system that estimates and quantifies a user's emotions based on text data and facial expression data.

[1098] "Emotion data" is user emotional information quantified by an emotion analysis engine.

[1099] "Means for optimizing the progress of meetings" refers to means for adjusting the progress of meetings and providing optimal facilitation based on quantified emotional data.

[1100] This invention relates to a system that automatically facilitates video conferences using AI. This system analyzes the content of participants' remarks, facial expressions, and tone of voice, and quantifies their emotions to provide optimal conference facilitation.

[1101] Hardware and software used

[1102] Hardware: high-resolution camera, microphone, server

[1103] Software: speech recognition software (e.g., Google Speech-to-Text), image recognition software (e.g., OpenCV), sentiment analysis engines (e.g., IBM Watson)

[1104] System Overview

[1105] 1. Acquiring user utterances and facial expressions

[1106] The device uses a high-resolution camera and microphone to capture what the user is saying and their facial expressions in real time.

[1107] Specifically, a camera captures the user's face and a microphone records the user's voice.

[1108] For example, if a user frowns while saying "this project is difficult," the camera and microphone will capture the moment.

[1109] 2. Data transmission

[1110] The terminal transmits the captured audio and video data to the server.

[1111] Specifically, the terminal divides the data into packets and sends them to a server over the Internet.

[1112] For example, data on the user's speech and facial expressions is sent to the server in real time.

[1113] 3. Voice and Image Recognition

[1114] The server uses voice recognition software to convert the voice data into text.

[1115] At the same time, the server uses image recognition software to analyze the user's facial expressions from the video data.

[1116] Specifically, speech recognition software analyzes speech waveforms and generates corresponding text, while image recognition software detects facial features and analyzes facial expressions.

[1117] For example, a statement such as "This project is difficult" is converted into text, and a frown is analyzed as "confusion."

[1118] 4. Quantifying emotions

[1119] The server uses an emotion analysis engine to estimate the user's emotions from text data and facial expression data and convert them into numerical values.

[1120] Specifically, the sentiment analysis engine takes text and facial expression data as input and generates an emotion score.

[1121] For example, a "confused" facial expression and a "difficult" statement will result in a high negative emotion score.

[1122] 5. Meeting Facilitation

[1123] The server optimizes the progress of the meeting based on the quantified emotional data.

[1124] Specifically, the server analyzes the emotion scores and, if necessary, pauses the meeting and suggests a break.

[1125] For example, if the emotion score of all participants is judged to be negative, the server will suggest, "Let's take a break."

[1126] Specific examples

[1127] For example, the following scenario is possible:

[1128] Scenario: During a meeting, many participants look tired and their comments become increasingly negative.

[1129] process:

[1130] The device's camera and microphone capture participants' facial expressions and speech.

[1131] The server uses voice recognition software to convert what is said into text and uses image recognition software to analyze facial expressions.

[1132] The emotion analysis engine quantifies emotions from text and facial expression data and determines that negative emotions are prevalent.

[1133] The server pauses the conference and suggests a break.

[1134] Prompt Sentence Examples

[1135] An example of a prompt sentence to input to the generative AI model is as follows:

[1136] "If many of the participants look tired and their comments become increasingly negative during the meeting, please suggest how to proceed."

[1137] By inputting this prompt into a generative AI model, the AI ​​will suggest appropriate ways to facilitate the meeting.

[1138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1139] Step 1:

[1140] The user speaks and shows facial expressions.

[1141] Input: User speech and facial expressions.

[1142] Specific behavior: A user says, "This project is difficult," and frowns.

[1143] Step 2:

[1144] The device uses a high-resolution camera and microphone to capture what the user is saying and their facial expressions in real time.

[1145] Input: User speech and facial expressions.

[1146] Output: Audio and video data.

[1147] What it does: The camera captures the user's face and the microphone records the user's voice.

[1148] Step 3:

[1149] The terminal transmits the captured audio and video data to the server.

[1150] Input: Audio and video data.

[1151] Output: Audio and video data sent to the server.

[1152] What it does: The device breaks the data into packets and sends them over the Internet to a server.

[1153] Step 4:

[1154] The server uses voice recognition software to convert the voice data into text.

[1155] Input: Audio data.

[1156] Output: Text data.

[1157] What it does: Speech recognition software analyzes the audio waveform and generates corresponding text.

[1158] Step 5:

[1159] The server uses image recognition software to analyze the user's facial expressions from the video data.

[1160] Input: Video data.

[1161] Output: Facial expression data.

[1162] What it does: Image recognition software detects facial features and analyzes facial expressions.

[1163] Step 6:

[1164] The server uses an emotion analysis engine to quantify the user's emotions from text data and facial expression data.

[1165] Input: Text data and facial expression data.

[1166] Output: Emotion data.

[1167] What it does: The sentiment analysis engine takes text and facial expression data as input and generates an emotion score.

[1168] Step 7:

[1169] The server optimizes the progress of the meeting based on the quantified emotional data.

[1170] Input: Emotion data.

[1171] Output: Instructions for running the meeting.

[1172] What it does: The server analyzes the emotion scores and pauses the meeting and suggests breaks as needed.

[1173] (Application example 1)

[1174] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1175] Conventional video conferencing systems and in-factory work support systems have had difficulty in properly analyzing the emotions of participants and workers and providing optimal facilitation and work support based on that analysis. Particularly during long work sessions and meetings, stress and fatigue accumulate among participants and workers, resulting in reduced efficiency and safety. This has created a demand for new methods to improve the quality of meetings and the efficiency and safety of work.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1177] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time and providing support for improving work efficiency and safety, means for suggesting breaks based on the emotion recognition results, and means for notifying the suggestion by voice using a speech synthesis engine, thereby making it possible to improve the efficiency and safety of meetings and work.

[1178] "Video conference facilitation" is the act of coordinating the proceedings and participants' comments during a video conference, and helping to achieve the meeting's objectives.

[1179] "AI automated means" refers to methods or devices that use artificial intelligence to automatically perform specific tasks.

[1180] "Means for analyzing the content of speech, facial expressions, and tone of voice of attendees and quantifying their emotions" refers to a method or device that uses voice recognition technology or image recognition technology to analyze the content of speech, facial expressions, and tone of voice of attendees and express the results as numerical values.

[1181] "Means for optimally facilitating a meeting" refers to a method or device for optimizing the progress of a meeting based on analyzed data.

[1182] "Means for analyzing worker emotions in real time and providing support to improve work efficiency and safety" refers to a method or device for analyzing worker emotions in real time and improving work efficiency and safety based on the results.

[1183] The "means for suggesting a break based on the result of emotion recognition" refers to a method or device for suggesting a break to workers or attendees based on the result of emotion recognition.

[1184] The "means for notifying a proposal by voice using a voice synthesis engine" refers to a method or device for notifying the proposal content by voice using voice synthesis technology.

[1185] The following system configuration is used as an embodiment of the present invention: The server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time and providing support to improve work efficiency and safety, means for suggesting breaks based on the emotion recognition results, and means for notifying suggestions by voice using a speech synthesis engine.

[1186] Hardware and software used

[1187] Hardware: Webcam, microphone

[1188] Software: OpenCV, Keras, pyttsx3

[1189] Data processing and calculation

[1190] The server captures video and audio in real time using a webcam and microphone. The captured video data undergoes facial recognition using OpenCV, and facial regions are extracted. The extracted facial regions are input into a Keras emotion recognition model, and emotions are quantified. Audio data is analyzed in the same way, and emotions are estimated from the tone of voice.

[1191] Based on the analyzed emotion data, the server will optimally facilitate the meeting. For example, if the emotions of all attendees are negative, the server will pause the meeting and suggest a break. Also, if a worker's emotions indicate stress or fatigue, the server will suggest a break. These suggestions are announced aloud using a speech synthesis engine (pyttsx3).

[1192] Specific examples

[1193] For example, consider a scenario in which an emotion recognition system detects the stress level of a worker who has been working for a long time in a factory and suggests that the worker take a break. In this case, the server uses a webcam and microphone to analyze the worker's facial expressions and tone of voice, and if it determines that the worker's stress level is high, it will issue a voice message saying, "Please take a break."

[1194] Example prompts to input to a generative AI model:

[1195] "Design a system that analyzes the emotions of workers in a factory in real time and suggests taking a break if stress or fatigue increases. Specifically, the system analyzes the facial expressions and tone of voice of workers and quantifies their emotions."

[1196] In this way, it is possible to improve the efficiency and safety of meetings and work.

[1197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1198] Step 1:

[1199] The server captures video and audio in real time using a webcam and microphone. The input is video data from the webcam and audio data from the microphone. The output is the captured raw video and audio data.

[1200] Step 2:

[1201] The server performs face recognition on the acquired video data using OpenCV. The input is the video data acquired in step 1. The output is image data with the face area cut out. Specifically, the position of the face is detected from the video data and that area is cut out.

[1202] Step 3:

[1203] The server inputs the extracted face region into the Keras emotion recognition model and quantifies the emotion. The input is the image data of the face extracted in step 2. The output is numerical data indicating the emotion. Specifically, the face image data is preprocessed and input into the emotion recognition model to predict the emotion.

[1204] Step 4:

[1205] The server analyzes the acquired voice data and estimates emotions from the tone of voice. The input is the voice data acquired in step 1. The output is numerical data indicating emotions based on the tone of voice. Specifically, the server preprocesses the voice data and extracts voice features to estimate emotions.

[1206] Step 5:

[1207] The server integrates the emotion data obtained in steps 3 and 4 to perform a comprehensive emotion evaluation. The input is facial emotion data and tone of voice emotion data. The output is numerical data indicating the comprehensive emotion evaluation. Specifically, the facial emotion data and tone of voice emotion data are integrated to calculate a comprehensive emotion score.

[1208] Step 6:

[1209] The server optimally facilitates the meeting and provides work support based on the overall emotional evaluation. The input is the overall emotional evaluation data obtained in step 5. The output is a specific action (e.g., suggesting a break). Specifically, if the emotional evaluation is negative, the server pauses the meeting and suggests a break.

[1210] Step 7:

[1211] The server uses a speech synthesis engine (pyttsx3) to notify the proposed content by voice. The input is the action determined in step 6. The output is a voice notification. Specifically, the break proposal is notified to attendees and workers by voice.

[1212] In this way, the server can analyze emotions in real time and provide optimal meeting facilitation and work support.

[1213] Example 2

[1214] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1215] In conventional video conferencing systems, meeting progress and facilitation are done manually, placing a heavy burden on attendees and reducing the quality and effectiveness of the meeting. It is also difficult to grasp attendees' emotions in real time and respond accordingly, which can impede smooth communication. Furthermore, when conducting specialized discussions, such as those focused on patents or strategies, it is difficult to generate appropriate questions and suggestions.

[1216] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for automatically facilitating the video conference using artificial intelligence, a means for analyzing the content of attendees' remarks, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for generating relevant questions and suggestions based on the conference theme and purpose, and a means for adjusting the progress of the conference based on emotion data. This reduces the burden on attendees and improves the quality and effectiveness of the conference. Furthermore, smooth communication can be achieved through real-time responses based on emotion data. Furthermore, even in specialized discussions such as those focused on patents or strategies, the depth and efficiency of discussions can be improved by automatically generating appropriate questions and suggestions.

[1217] A "video conference" is a type of conference in which multiple participants communicate in real time using audio and video over the Internet.

[1218] "Facilitation" is the process of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[1219] "Artificial intelligence" refers to computer systems that mimic human intelligence and can learn, reason, and solve problems.

[1220] "Speech" refers to information and opinions verbally expressed by participants during a meeting.

[1221] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[1222] "Vocal timbre" refers to characteristics of speech, such as tone, pitch, and rhythm, and is an element that conveys the speaker's emotions and intentions.

[1223] "Quantifying emotions" means expressing analyzed emotional data as quantitative numbers.

[1224] "Optimal meeting facilitation" means conducting and adjusting the meeting in the most effective manner depending on the purpose and situation of the meeting.

[1225] A "conference theme" is the main topic or issue to be discussed at a conference.

[1226] "Purpose" refers to the specific goal or outcome that the meeting aims to achieve.

[1227] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on the theme and purpose of the meeting.

[1228] "Emotional data" refers to analytical results that indicate the emotional state of participants.

[1229] "Coordinating the progress of a meeting" means properly managing the flow and progress of the meeting and making adjustments to ensure that it proceeds smoothly.

[1230] "Patent-specific" is a conference format specialized for discussing and examining patent-related matters.

[1231] "Strategy-focused" is a meeting format specialized for discussing and examining strategy-related matters.

[1232] MODE FOR CARRYING OUT THE INVENTION

[1233] This invention is a system that uses artificial intelligence to automatically facilitate video conferences, and has the ability to analyze attendees' comments, facial expressions, and tone of voice to quantify their emotions. It also has the ability to generate relevant questions and suggestions based on the theme and purpose of the conference, and adjust the progress of the conference based on emotional data.

[1234] System configuration

[1235] This system consists of three main components: a server, a terminal, and a user.

[1236] server

[1237] The server uses a generative AI model to generate questions and suggestions based on the meeting's theme and purpose. For example, OpenAI's GPT-4 is used as a generative AI model. The server receives input from the user, generates appropriate questions and suggestions based on that information, and sends them to the device. The server also receives emotion data sent from the device and determines actions to take to adjust the progress of the meeting.

[1238] Terminal

[1239] The device uses a camera and microphone to detect the user's emotions in real time. The emotion engine uses, for example, the Emotion API from Microsoft's Azure Cognitive Services. The device captures the user's facial expressions through the camera and records the tone of voice through the microphone. This data is sent to the emotion engine, which analyzes the user's emotions in real time. The device receives instructions from the server and adjusts the progress of the meeting.

[1240] User

[1241] Users use their devices to input the topic and purpose of the meeting, for example, "I want to discuss new patent ideas." If a user expresses anger or frustration during the meeting, the device sends that information to the server, which then decides what action to take to adjust the progress of the meeting.

[1242] Specific examples

[1243] As a concrete example, consider the case of patent-specific facilitation. When a user inputs, "I would like to discuss a new patent idea," the server uses a generative AI model to automatically generate questions such as, "What is the technical novelty of this idea?" and "How is it different from existing patents?"

[1244] As a concrete example of an emotion engine, consider the case where a participant expresses dissatisfaction during a meeting. The device analyzes the participant's facial expression and tone of voice to detect dissatisfaction. The server receives this information and instructs the device to provide an opportunity for the participant to speak, such as, "Mr. / Ms. XX, do you have any comments?"

[1245] Prompt Sentence Examples

[1246] "I'd like to discuss new patent ideas."

[1247] "Generate questions about the novelty of this technology."

[1248] "Please suggest appropriate responses to participants who express dissatisfaction during a meeting."

[1249] In this way, the system can reduce the burden on users and improve the quality and effectiveness of meetings.

[1250] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1251] Step 1:

[1252] The user inputs the theme and purpose of the meeting. The user uses the keyboard on the device to type, "I would like to discuss new patent ideas." This input data is sent from the device to the server.

[1253] Step 2:

[1254] The server uses a generative AI model to generate questions and suggestions related to the conference theme. The server inputs the theme and purpose received from the user into the generative AI model, generating questions such as "What is the technical novelty of this idea?" and "How is it different from existing patents?" The generated questions are sent from the server to the terminal.

[1255] Step 3:

[1256] The device detects the user's emotions in real time. The device captures the user's facial expressions through the camera and records the tone of voice through the microphone. This data is sent to the emotion engine, which analyzes the user's emotions in real time. For example, if the user frowns, the emotion of anger is detected. This emotion data is sent from the device to the server.

[1257] Step 4:

[1258] The server receives the emotion data and determines an action to adjust the progress of the conference. The server receives the emotion data sent from the terminal and determines an action to adjust the progress of the conference based on that data. For example, if a user expresses dissatisfaction, the server instructs the user to be given an opportunity to speak. This instruction is sent from the server to the terminal.

[1259] Step 5:

[1260] The terminal follows instructions from the server and adjusts the progress of the conference. The terminal receives instructions from the server and displays a message to the user asking, "Mr. / Ms. XX, do you have any comments?" When a user begins to speak, the terminal sends the content to the server, ensuring the conference proceeds smoothly.

[1261] (Application example 2)

[1262] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1263] With conventional video conferencing systems, it was difficult to detect participants' emotions in real time and adjust the progress of the meeting based on that. Furthermore, there was a lack of means to provide specialized facilitation for patents or strategies, making it difficult to improve the quality and effectiveness of meetings. Furthermore, there was a need for a way to properly manage participants' emotions and promote smooth communication in factory meetings and brainstorming sessions.

[1264] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1265] In this invention, the server includes a means for automatically facilitating video conferences using AI, a means for analyzing participants' comments, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for supporting factory meetings and brainstorming sessions, and a means for detecting participants' emotions in real time using an emotion engine and adjusting the progress of the conference. This makes it possible to detect participants' emotions in real time and adjust the progress of the conference based on that information, providing specialized facilitation for patents and strategies. It also makes it possible to appropriately manage participants' emotions in factory meetings and brainstorming sessions, promoting smooth communication.

[1266] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[1267] "Facilitation" is the activity of helping meetings and group discussions proceed smoothly and ensuring that all participants can exchange opinions effectively.

[1268] "AI" stands for artificial intelligence, a technology that allows computers to mimic human intelligence and perform functions such as learning, reasoning, and recognition.

[1269] "Speech content" refers to opinions and information verbally expressed by participants in meetings and discussions.

[1270] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[1271] "Vocal timbre" refers to the characteristics of a speaker's voice, such as timbre, tone, and rhythm, and is an element that conveys emotions and intentions.

[1272] "Quantifying emotions" means analyzing participants' emotional states and expressing them as numerical data.

[1273] A "factory meeting" is a meeting held at a manufacturing site, where discussions are held on product production, quality control, business improvements, and so on.

[1274] A "brainstorming session" is a discussion format in which participants freely exchange ideas and find creative solutions.

[1275] The "emotion engine" is a technology that analyzes data such as participants' facial expressions and tone of voice to detect their emotional state in real time.

[1276] "Real-time detection" means analyzing data instantly and obtaining results without delay.

[1277] "Coordinating the progress of a meeting" means managing the flow and progress of the meeting and intervening or correcting as necessary.

[1278] "Patent-specific facilitation" is a specialized support activity to promote patent-related discussions.

[1279] "Strategic-focused facilitation" is a specialized support activity aimed at promoting discussions on corporate and organizational strategies.

[1280] A system for implementing this invention includes means for automatically facilitating video conferences using AI, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating meetings, means for supporting meetings and brainstorming sessions within factories, and means for using an emotion engine to detect participants' emotions in real time and adjust the progress of the meeting.

[1281] The server first uses a camera (e.g., Logitech C920) to capture video of meeting participants in real time. Next, it uses an emotion detection engine called EmotionEngine to analyze the participants' emotions from the captured video and express them as numerical data. The analyzed emotional data is sent to a facilitation support AI called FacilitationAI, which suggests actions to optimize the progress of the meeting.

[1282] Specifically, if one participant shows anger or dissatisfaction, FacilitationAI instructs the server to provide that participant with an opportunity to speak. Furthermore, if a participant shows joy or satisfaction, it suggests actions to encourage further participation. This helps the meeting proceed smoothly and improves the satisfaction of all participants.

[1283] For example, if a meeting is being held at a factory to discuss the introduction of a new product line and one participant expresses dissatisfaction, the server can give that participant an opportunity to speak and convey their feelings to the other participants, thereby smoothing the meeting.

[1284] Examples of prompts for generative AI models include:

[1285] "We are having a meeting about introducing a new product line. One of the participants is expressing frustration. Please suggest what facilitation action should be taken in this situation."

[1286] In this way, the server can detect participants' emotions in real time and adjust the progress of the meeting based on that information, providing specialized facilitation for patents and strategies, and even in factory meetings and brainstorming sessions, it can properly manage participants' emotions and promote smooth communication.

[1287] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1288] Step 1:

[1289] The server captures video of the meeting participants in real time using a camera (e.g., Logitech C920). The input is the video data from the camera, and the output is the captured video frames, which are used in subsequent processing steps.

[1290] Step 2:

[1291] The server uses EmotionEngine to analyze participants' facial expressions from captured video frames and quantify their emotions. The input is the video frames, and the output is numerical data that indicates the emotional state of each participant. Specifically, EmotionEngine analyzes the video frames, detects facial feature points, and classifies emotions.

[1292] Step 3:

[1293] The server sends the analyzed emotional data to Facilitation AI, which then proposes actions to optimize the progress of the meeting. The input is emotional data, and the output is proposed facilitation actions. Specifically, Facilitation AI analyzes the emotional data and determines appropriate actions (e.g., providing an opportunity to speak, encouraging participation).

[1294] Step 4:

[1295] The server executes the actions proposed by FacilitationAI. The input is the proposed action, and the output is the result of the executed action. Specifically, the server coordinates the progress of the meeting, providing participants with opportunities to speak as needed and communicating their emotional state to other participants.

[1296] Step 5:

[1297] The server monitors the progress of the meeting and, if necessary, uses the Emotion Engine again to detect participants' emotions and send them to FacilitationAI. The input is the progress of the meeting and the latest video frames, and the output is updated emotional data and new facilitation actions. Specifically, the server periodically captures video frames and re-analyzes emotions to optimize the progress of the meeting.

[1298] Step 6:

[1299] After the meeting, the server saves all emotional data and the history of facilitation actions to help improve subsequent meetings. The input is the emotional data and the action history, and the output is the saved data. Specifically, the server saves the emotional data and the action history in a database and uses them as reference material for the next meeting.

[1300] Example 3

[1301] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1302] With conventional video conferencing systems, it was difficult to grasp the emotional state of attendees in real time and adjust the progress of the meeting appropriately. Furthermore, it was not possible to predict changes in attendees' emotions and optimize the progress of the meeting based on those predictions, which could result in a decline in the quality and effectiveness of the meeting. Furthermore, non-face-to-face communication often did not proceed smoothly, which increased the burden on users.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1304] In this invention, the server includes means for collecting attendee data, means for analyzing the collected data and evaluating the attendees' emotional states, means for adjusting the progress of the meeting based on the evaluation results, means for predicting attendees' emotional changes and optimizing the progress of the meeting based on the prediction, and means for providing smooth communication even when not face-to-face. This makes it possible to grasp the attendees' emotional states in real time and automatically provide appropriate facilitation. Furthermore, by predicting attendees' emotional changes and optimizing the progress of the meeting, the quality and effectiveness of the meeting can be improved. Furthermore, by providing smooth communication even when not face-to-face, the burden on users can be reduced.

[1305] "Means for collecting attendee data" refers to devices or software for collecting data such as facial expressions and voices of attendees in a meeting in real time.

[1306] The "means for analyzing the collected data and assessing the emotional state of attendees" refers to algorithms or software for analyzing the collected facial and voice data and assessing the emotional state of attendees (such as joy, anger, sadness, etc.).

[1307] The "means for adjusting the progress of the meeting based on the evaluation results" refers to a system or process for pausing the progress of the meeting or suggesting a break based on the evaluation results of the emotional state.

[1308] "Means for predicting changes in the emotions of attendees and optimizing the progress of the meeting based on the predictions" refers to algorithms or software that predict changes in the emotions of attendees based on past emotional data and adjust the progress of the meeting based on the prediction results.

[1309] "Means for providing smooth communication even when not face-to-face" refers to systems and processes that enable attendees to communicate smoothly with each other even when they are not physically face-to-face.

[1310] This invention is a system for facilitating non-face-to-face communication, and includes an AI function for automatically adjusting the progress of a meeting. Specific embodiments of this system are described below.

[1311] As soon as the meeting starts, the server collects facial expression and voice data from attendees via cameras and microphones connected to their devices (PCs or smartphones). The devices then send this data to the server in real time. The hardware used includes high-resolution cameras, microphones, and servers. The software used includes an emotion analysis engine (e.g., Microsoft Azure's Emotion API) and a meeting management system (e.g., Zoom or Microsoft Teams).

[1312] The server sends the collected facial and voice data to an emotion analysis engine, which analyzes the data and evaluates the attendees' emotional state (happiness, anger, sadness, surprise, etc.) in real time. For example, if an attendee's facial expression becomes grim and their voice tone becomes lower, the emotion analysis engine will determine that the attendee is expressing negative emotions. Based on this information, the server pauses the meeting and suggests that attendees take a break.

[1313] Furthermore, the emotion engine predicts changes in attendees' emotions based on past data. For example, if it predicts that a certain attendee is likely to be dissatisfied with the progress of the meeting, the server will provide that attendee with an opportunity to speak in advance, or take other action such as adjusting the progress of the meeting.

[1314] To illustrate, consider the following scenario:

[1315] Scenario: Participant A becomes frustrated during a meeting.

[1316] Processing: The emotion analysis engine analyzes Attendee A's facial expressions and tone of voice and detects dissatisfaction. The server pauses the meeting and gives Attendee A an opportunity to speak.

[1317] Example prompts to input to a generative AI model:

[1318] "Design a system that analyzes attendees' emotions in real time during a meeting, and if negative emotions are detected, pause the meeting and suggest a break. Also, add a function that predicts changes in attendees' emotions and optimizes the progress of the meeting."

[1319] In this way, the server analyzes the emotions of the attendees and automatically performs appropriate facilitation, thereby realizing an environment that provides smooth communication even without face-to-face communication. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1320] Step 1:

[1321] The server activates the camera and microphone connected to the device as soon as the meeting starts. The device captures the facial expressions and audio data of the participants in real time and sends them to the server. The input is the video and audio data from the camera and microphone, and the output is the raw data sent to the server.

[1322] Step 2:

[1323] The server sends the received facial expression and voice data to the emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the emotional state of the attendees. The input is the raw data sent from the server, and the output is the evaluation result of the emotional state (happiness, anger, sadness, surprise, etc.). Specifically, the emotion analysis engine analyzes facial expression features (e.g., eyebrow movement, mouth shape) and voice tone.

[1324] Step 3:

[1325] The server receives the evaluation results of the emotional state returned from the emotion analysis engine and determines whether the attendees' emotions are positive or negative. The input is the evaluation result from the emotion analysis engine, and the output is the judgment result of the emotional state. Specifically, if the server determines that the attendees' emotions are negative based on the evaluation results, it pauses the progress of the meeting.

[1326] Step 4:

[1327] If a negative emotion is detected, the server instructs the device to display a message suggesting a break. The input is the result of the emotional state judgment, and the output is the break suggestion message displayed on the device. Specifically, the server sends a message to the device such as, "The participant's emotions are leaning negative, so we suggest a 5-minute break."

[1328] Step 5:

[1329] The server predicts changes in the emotions of attendees based on past emotion data. The input is past emotion data, and the output is the predicted emotion change. Specifically, the emotion engine uses a machine learning algorithm to predict whether a particular attendee is likely to be dissatisfied with the progress of the meeting.

[1330] Step 6:

[1331] The server takes action to optimize the progress of the meeting based on the prediction of emotion changes. The input is the predicted emotion change result, and the output is instructions for the optimized progress of the meeting. Specifically, the server provides attendees who are predicted to be dissatisfied with the meeting with an opportunity to speak in advance. It also takes action such as adjusting the progress of the meeting.

[1332] In this way, the server analyzes the emotions of attendees and automatically provides appropriate facilitation, creating an environment that provides smooth communication even when not face-to-face.

[1333] (Application example 3)

[1334] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1335] With conventional video conferencing systems and customer service systems, it was difficult to properly analyze the emotions of attendees and customers and adjust responses in real time based on that analysis. As a result, meetings could not proceed smoothly, leading to lower participant satisfaction, and customer service could be inadequate, leading to lower customer satisfaction. There is a need to solve these issues and improve the quality of meetings and customer service.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1337] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for analyzing customers' facial expressions and tone of voice and determining their emotions in real time, means for proposing appropriate responses based on the results of the emotion analysis, and means installed on a smart device to support customer service. This makes it possible for the conference to proceed smoothly, improving the satisfaction of all participants, and for customer service to be provided quickly and appropriately, thereby improving customer satisfaction.

[1338] "Video conference facilitation" refers to the coordination and management of video conferences to ensure their smooth progress.

[1339] "Automated AI means" refers to methods or devices that allow artificial intelligence to automatically perform specific tasks.

[1340] "Means for analyzing the remarks, facial expressions, and tone of voice of attendees and quantifying their emotions" refers to a method or device for analyzing the remarks, facial expressions, and tone of voice of meeting participants and expressing their emotions as a number.

[1341] "Means for optimal meeting facilitation" refers to methods and devices for optimizing the progress of meetings.

[1342] "Means for analyzing a customer's facial expression and tone of voice and determining their emotions in real time" refers to a method or device for analyzing a customer's facial expression and tone of voice and determining their emotions in real time.

[1343] The term "means for proposing an appropriate response based on the result of emotion analysis" refers to a method or device for proposing an appropriate response based on the result of emotion analysis.

[1344] "Means installed on a smart device to assist in customer service" refers to a method or apparatus that is installed on a device such as a smartphone or smart glasses to assist in customer service.

[1345] This invention is a system that uses AI to analyze emotions during video conferences and customer service, and proposes appropriate responses in real time. The system has functions to automatically facilitate the smooth progress of video conferences and to support customer service.

[1346] System configuration

[1347] The system consists of the following major components:

[1348] 1. Video conference facilitation module: AI automatically manages video conferences.

[1349] 2. Emotion analysis module: Analyzes the content, facial expressions, and tone of voice of attendees and customers, and quantifies their emotions.

[1350] 3. Response suggestion module: Based on the results of sentiment analysis, it suggests optimal meeting facilitation and customer response.

[1351] 4. Smart device interface: Installed on devices such as smartphones and smart glasses, it assists in customer service.

[1352] Hardware and software used

[1353] Hardware: smart glasses, smartphone, camera, microphone

[1354] Software: OpenCV (face detection), Keras (emotion analysis model)

[1355] Data processing and calculation

[1356] 1. Face detection: Capture the faces of customers and attendees in real time using smart glasses or smartphone cameras. Use OpenCV to detect faces and obtain facial expression data.

[1357] 2. Emotion analysis: The acquired facial expression data is input into the Keras emotion analysis model to quantify emotions, thereby determining the emotional state of customers and attendees in real time.

[1358] 3. Response suggestions: Based on the results of emotion analysis, appropriate responses are suggested. For example, if a customer is dissatisfied, the smart glasses display will say, "The customer is dissatisfied. Please improve your response."

[1359] Specific examples

[1360] scenario

[1361] If a store clerk is interacting with a customer and the customer looks dissatisfied, the smart glasses' display will display a message saying, "The customer is dissatisfied. Please improve your service." This allows the store clerk to quickly improve their service and increase customer satisfaction.

[1362] Prompt Sentence Examples

[1363] "Capture the customer's facial expressions and use an emotion analysis model to determine their emotions. If the customer is dissatisfied, display a message suggesting how to improve the experience."

[1364] In this way, a system that supports customer service in video conferences and brick-and-mortar stores can be realized. This system will enable meetings to proceed smoothly, improving the satisfaction of all participants, and enabling customers to be served promptly and appropriately, thereby improving customer satisfaction.

[1365] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1366] Step 1:

[1367] Face detection

[1368] The server captures the faces of customers and attendees in real time using smart glasses or a smartphone camera. As input, it receives video data from the camera and detects faces using OpenCV. As output, it obtains the position information and facial expression data of the detected faces. Specifically, it analyzes the camera video frame by frame and identifies the facial contours.

[1369] Step 2:

[1370] Emotion analysis

[1371] The server inputs the facial expression data obtained in step 1 into the Keras emotion analysis model. It receives facial position information and facial expression data as input and quantifies emotions using the emotion analysis model. The output is the emotional state of the customer or attendee (e.g., joy, sadness, anger, etc.). Specifically, it preprocesses the facial expression data and inputs it into the emotion analysis model to predict emotions.

[1372] Step 3:

[1373] Response proposal

[1374] The server proposes an appropriate response based on the emotion analysis results obtained in step 2. It receives the emotional state as input and determines the appropriate response using the response suggestion module. As output, it generates a message to be displayed on the display of the smart glasses or smartphone. Specifically, it searches a database for a response appropriate to the emotional state and generates a response message.

[1375] Step 4:

[1376] Message display

[1377] The device displays the corresponding message generated in step 3 on the display of the smart glasses or smartphone. As input, it receives the corresponding message and processes it to display it on the display. As output, it visually presents the message to the user. Specifically, it calls the display's API and displays the message.

[1378] Step 5:

[1379] User response

[1380] The user responds appropriately to the customer or attendees based on the message displayed in step 4. As input, the user receives the displayed message and implements a countermeasure. As output, the user improves the satisfaction of the customer or attendees. As a specific action, the user adjusts the customer response or the progress of the meeting according to the message.

[1381] In this way, the system can analyze emotions in real time and suggest appropriate responses, improving the quality of meetings and customer interactions.

[1382] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1385] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1386] [Third embodiment]

[1387] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1388] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1389] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1390] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1391] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1393] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1394] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1397] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1398] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1399] "Example 1"

[1400] One aspect of the present invention is a system that uses AI to automatically facilitate video conferences. Specifically, it is equipped with an AI engine that analyzes the content, facial expressions, and tone of voice of attendees. This AI engine uses voice recognition and image recognition technology to quantify attendees' emotions, and based on the results, optimally facilitates the conference.

[1401] "Example 2"

[1402] Furthermore, it also has the ability to provide specialized facilitation, such as patent-specific and strategy-specific facilitation. For example, in patent-specific facilitation, AI automatically asks questions and makes suggestions to promote patent-related discussions. This reduces the burden on users and improves the quality and effectiveness of meetings.

[1403] "Example 3"

[1404] It also has a function that provides an environment where smooth communication can be carried out even without face-to-face meetings. Specifically, the AI ​​analyzes the emotions of attendees and adjusts the progress of the meeting based on the results. For example, if it determines that attendees' emotions are leaning negatively, the AI ​​can automatically perform appropriate facilitation like a human would, such as pausing the meeting and suggesting that attendees take a break.

[1405] The processing flow of each embodiment will be described below.

[1406] "Example 1"

[1407] Step 1: The AI ​​engine starts the video conference and analyzes the participants' speech, facial expressions, and tone of voice in real time.

[1408] Step 2: The AI ​​engine uses voice and image recognition technology to quantify the emotions of attendees.

[1409] Step 3: The AI ​​engine uses the analysis results to optimally facilitate the meeting. For example, if it determines that attendees' emotions are leaning negatively, it will pause the meeting and suggest that attendees take a break.

[1410] "Example 2"

[1411] Step 1: The AI ​​engine provides specialized facilitation, such as patent-specific or strategy-specific facilitation.

[1412] Step 2: In patent-specific facilitation, AI automatically asks questions and makes suggestions to promote patent-related discussions.

[1413] Step 3: This reduces the burden on users and improves the quality and effectiveness of meetings.

[1414] "Example 3"

[1415] Step 1: Provide an environment where the AI ​​engine can communicate smoothly even without face-to-face contact.

[1416] Step 2: AI analyzes attendees' emotions and adjusts the meeting progress based on the results.

[1417] Step 3: For example, if the AI ​​determines that attendees' emotions are leaning negatively, it can automatically take appropriate facilitation steps, just like a human would, such as pausing the meeting and suggesting that attendees take a break.

[1418] Example 1

[1419] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1420] Conventional video conferencing systems have difficulty in properly analyzing the content, facial expressions, and tone of voice of attendees and optimizing the progress of meetings. Furthermore, there was a lack of means to provide feedback to improve the quality and effectiveness of meetings, which placed a heavy burden on users and hindered smooth non-face-to-face communication.

[1421] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1422] In this invention, the server includes a means for automatically facilitating the video conference using artificial intelligence, a means for analyzing the content of attendees' remarks, facial expressions, and tone of voice and quantifying their emotions, a means for converting voice data into text using speech recognition technology, a means for analyzing facial expressions from video data using image recognition technology, a means for optimal conference facilitation based on the quantified emotional data, and a means for providing feedback based on the progress of the conference and the emotional data. This makes it possible to analyze attendees' emotions in real time and ensure optimal conference progress. Furthermore, by providing feedback to improve the quality and effectiveness of the conference, the burden on users is reduced and smooth communication can be achieved even when not face-to-face.

[1423] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[1424] "Facilitation" is a technique for smoothly progressing meetings and group discussions and enabling participants to exchange opinions effectively.

[1425] "Artificial intelligence" refers to a computer system that mimics human intelligence and has capabilities such as learning, reasoning, and recognition.

[1426] "Speech" refers to the words and opinions expressed orally by participants during a meeting.

[1427] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[1428] "Vocal timbre" refers to characteristics such as the quality of the voice, tone, and intonation, and is an element that conveys the speaker's emotions and intentions.

[1429] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[1430] "Quantifying emotions" means quantitatively evaluating emotions and expressing them as numbers.

[1431] "Speech recognition technology" is a technology that converts speech into text, and can automatically record what a speaker says as text.

[1432] "Image recognition technology" is a technology that analyzes image data and recognizes specific patterns and features.

[1433] "Optimal meeting facilitation" is a method for conducting a meeting most effectively, taking into account the circumstances and emotions of the participants.

[1434] "Feedback" refers to evaluations and advice provided based on the progress and results of the meeting.

[1435] "Progress" indicates how the meeting is progressing and its current status.

[1436] "Emotional data" is data that quantifies the emotions of participants and can be used to guide and facilitate meetings.

[1437] This invention relates to a system that uses artificial intelligence to automatically facilitate video conferences. This system analyzes the content of participants' remarks, facial expressions, and tone of voice, and quantifies their emotions to achieve optimal conference facilitation.

[1438] Hardware and software used

[1439] Hardware: Camera, microphone, device (PC, tablet, smartphone)

[1440] Software: speech recognition technology (e.g., Google Speech-to-Text), image recognition technology (e.g., OpenCV), AI engine (e.g., TensorFlow)

[1441] System Operation

[1442] Subject: Server

[1443] The server provides a system that automatically facilitates video conferences. This system is equipped with an AI engine that analyzes the content of participants' remarks, facial expressions, and tone of voice. The server uses voice recognition and image recognition technology to quantify the emotions of participants. Based on this quantified emotional data, the server provides optimal conference facilitation. It also provides feedback based on the progress of the conference and the emotional data.

[1444] Subject: Terminal

[1445] The device transmits the video and audio of the video conference to the server. The device also uses a camera and microphone to capture the facial expressions and speech of participants. This data is sent to the server in real time and analyzed by an AI engine.

[1446] Subject: User

[1447] Users participate in video conferences and send their comments and facial expressions to the server via their devices. Users simply participate in the conference in a natural way, without any special operations. The AI ​​engine analyzes the content of the user's comments, facial expressions, and tone of voice, providing optimal conference facilitation.

[1448] Specific examples

[1449] Example 1: Analysis of speech during a meeting

[1450] When User A says, "How is the progress on this project?", the server uses voice recognition technology to convert the speech into text. The AI ​​engine then analyzes this text and recognizes that User A is asking a question. The server then facilitates the meeting by encouraging other attendees to provide appropriate answers.

[1451] Example 2: Facial expression analysis

[1452] The device's camera captures User B's facial expression and sends it to the server. The server uses image recognition technology to analyze User B's facial expression and detects that User B is confused. The server pauses the meeting and facilitates by asking User B questions or opinions.

[1453] Prompt Sentence Examples

[1454] "Please describe a system that analyzes what participants say, their facial expressions, and tone of voice during a video conference to provide optimal meeting facilitation. Please include the names of any specific hardware or software."

[1455] In this way, the system's operation can be explained from the perspectives of the server, the device, and the user, and concrete examples are used to deepen understanding. This system can analyze attendees' emotions in real time and ensure optimal meeting progress. It also provides feedback to improve the quality and effectiveness of meetings, reducing the burden on users and enabling smooth communication even when not face-to-face.

[1456] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1457] Step 1: Start a video conference

[1458] Subject: User

[1459] A user starts a video conference using a terminal. The user starts a conference application and enters the conference ID to join. For example, when a user joins a conference using a conference application, the user enters the conference ID and clicks the "Join" button.

[1460] Input: Meeting ID, user operation

[1461] Output: Start a video conference and prepare to capture video and audio

[1462] Step 2: Capturing Data

[1463] Subject: Terminal

[1464] The device uses a camera and microphone to capture the user's video and audio. The camera captures the user's facial expressions in real time, and the microphone records what the user says. For example, the device's camera captures the user's smile, and the microphone records when the user says "hello."

[1465] Input: User video and audio

[1466] Output: Captured video and audio data

[1467] Step 3: Sending data

[1468] Subject: Terminal

[1469] The device sends the captured video and audio data to the server. The data is encrypted and sent securely to the server. For example, the device sends the captured video and audio data to the server using the HTTPS protocol.

[1470] Input: Captured video and audio data

[1471] Output: Data sent to the server

[1472] Step 4: Analyze the data

[1473] Subject: Server

[1474] The server analyzes the received video and audio data. It uses voice recognition technology to convert the audio data into text and image recognition technology to analyze facial expressions from the video data. For example, the server uses voice recognition technology to convert the audio of "hello" into text and image recognition technology to detect the user's smile.

[1475] Input: Video and audio data sent to the server

[1476] Output: Text data, facial expression analysis results

[1477] Step 5: Quantify emotions

[1478] Subject: Server

[1479] The server quantifies the user's emotions based on the analysis results. The AI ​​engine analyzes changes in voice tone and facial expressions to generate an emotion score. For example, the server quantifies the user's smile as "joy" and assigns an emotion score of 80 / 100.

[1480] Input: Text data, facial expression analysis results

[1481] Output: Sentiment score

[1482] Step 6: Facilitation

[1483] Subject: Server

[1484] The server then optimally facilitates the meeting based on the quantified emotional data. For example, if a user is confused, the server may pause the meeting and prompt the user to ask questions or give their opinions.

[1485] Input: Sentiment score

[1486] Output: Facilitation instructions

[1487] Step 7: Provide feedback

[1488] Subject: Server

[1489] The server provides feedback based on the progress of the meeting and emotional data. For example, after the meeting, it sends a report to attendees about changes in their emotions and the frequency of their comments during the meeting.

[1490] Input: Meeting progress, emotion data

[1491] Output: Feedback report

[1492] In this way, by explaining each processing step specifically and clearly showing the inputs and outputs, the operation of the system can be more clearly understood.

[1493] (Application example 1)

[1494] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1495] With conventional video conferencing systems, it was difficult to analyze participants' remarks, facial expressions, and tone of voice to quantify their emotions and provide optimal meeting facilitation. Furthermore, while efficient progress is required for factory meetings and discussions, there was a lack of automated means for conducting meetings that took participants' emotions into consideration. This resulted in a decline in the quality and effectiveness of meetings and an increased burden on users.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1497] In this invention, the server includes means for automatically faciliting video conferences using AI, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating the conference, means to be installed in a robot for streamlining in-plant meetings and discussions, and means for optimizing the progress of the conference taking into account the emotions of attendees.This makes it possible to efficiently conduct in-plant meetings and discussions while taking into account the emotions of attendees.

[1498] A "video conference" is a conference in which multiple participants share video and audio in real time via the Internet.

[1499] "Facilitation" refers to methods and techniques for smoothly conducting meetings and discussions.

[1500] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to learn and reason by imitating human intelligence.

[1501] "Speech content" refers to the information and opinions that participants verbally express during a meeting or discussion.

[1502] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[1503] "Vocal timbre" refers to the tone and texture of a voice, and is an element that reflects the speaker's emotions and intentions.

[1504] "Quantifying emotions" means analyzing participants' emotional states and expressing them as numerical data.

[1505] "Optimal meeting facilitation" means taking into consideration the content and emotional state of participants and conducting the meeting in the most effective manner.

[1506] A "factory meeting" is a meeting held to discuss matters related to factory operations and production.

[1507] A "meeting" is when multiple people gather to discuss a specific purpose or issue and exchange opinions and share information.

[1508] A "robot" is a mechanical device that operates automatically according to a program to perform specific tasks.

[1509] "Installation" means introducing software or applications into a computer or machine device and making them ready for use.

[1510] "Optimizing the progress of a meeting" means making the necessary adjustments and improvements to ensure that the meeting proceeds efficiently and effectively.

[1511] As an embodiment of this invention, we will explain a "smart conference robot" system for streamlining meetings and discussions within factories. This system uses AI to automatically facilitate video conferences, analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions and provide optimal conference facilitation.

[1512] System configuration

[1513] 1. Hardware:

[1514] Camera: Used to capture attendees' facial expressions in real time.

[1515] Microphone: Used to capture what attendees are saying.

[1516] Robot: A mechanical device that moves around the factory and helps run meetings.

[1517] 2. Software:

[1518] Speech Recognition Library: Use the speech_recognition library to recognize what attendees are saying in real time.

[1519] Emotion analysis model: Using the transformers library pipeline, we analyze attendees' facial expressions and tone of voice to quantify their emotions.

[1520] Image processing library: Uses the cv2 library to acquire images from the camera in real time and perform facial expression analysis.

[1521] System Operation

[1522] 1. Speech Recognition:

[1523] The server converts the audio data acquired from the microphone into text data using the speech_recognition library.

[1524] The converted text data is used as information for optimizing the progress of the conference.

[1525] 2. Emotion analysis:

[1526] The server processes the video data acquired from the camera in real time using the cv2 library and analyzes the facial expressions of the attendees.

[1527] The analyzed facial expression data is used to quantify emotions using a pipeline from the transformers library.

[1528] 3. Optimizing meeting progress:

[1529] The server executes logic to optimize the progress of the conference based on the acquired voice data and emotion data.

[1530] For example, if an attendee says, "There's a problem with this part," the robot will respond, "There's a problem. We'll suggest a solution."

[1531] Also, if the attendee's facial expression is negative, the message "The attendee's emotions are negative. We will adjust the meeting progress." will be displayed.

[1532] Specific examples

[1533] As a concrete example, consider a quality control meeting in a factory. If a participant says, "There is a problem with this part," the robot responds, "A problem has occurred. We will propose a solution." If the participant's facial expression is negative, the robot displays, "The participant's emotions are negative. We will adjust the progress of the meeting."

[1534] Prompt Sentence Examples

[1535] Examples of prompts to input to a generative AI model include:

[1536] If someone says, "We have a problem with this part," please suggest how the meeting should proceed.

[1537] In this way, the smart conference robot system can make meetings within factories more efficient and provide optimal facilitation that takes into account the emotions of attendees.

[1538] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1539] Step 1:

[1540] The server acquires video data from the camera in real time. The input is video data from the camera, and the output is frame data processed using the image processing library cv2. Specifically, the server captures the video from the camera frame by frame, converts it to grayscale, and performs face detection.

[1541] Step 2:

[1542] The server acquires voice data from the microphone in real time. The input is the voice data from the microphone, and the output is the speech content converted into text data using the speech recognition library, speech_recognition. Specifically, the server records the voice from the microphone and converts it into text using a speech recognition engine.

[1543] Step 3:

[1544] The server analyzes the facial expressions of attendees using the acquired frame data. The input is the frame data obtained in step 1, and the output is emotional data quantified using the emotion analysis model transformers pipeline. Specifically, the server extracts the area where the face was detected, inputs it into the emotion analysis model, and quantifies the emotion.

[1545] Step 4:

[1546] The server analyzes the acquired text data and extracts information necessary for the progress of the meeting. The input is the text data obtained in step 2, and the output is instructions and suggestions for the progress of the meeting. Specifically, the server inputs the text data into a natural language processing engine and extracts important keywords and phrases.

[1547] Step 5:

[1548] The server executes logic to optimize the progress of the meeting based on the emotion data and text data. The input is the emotion data obtained in step 3 and the text data obtained in step 4, and the output is specific actions for the progress of the meeting. As a specific operation, the server integrates the emotion data and text data and generates instructions for adjusting the progress of the meeting.

[1549] Step 6:

[1550] The server sends the generated instructions to the robot to support the progress of the meeting. The input is the instructions for proceeding with the meeting obtained in step 5, and the output is specific actions by the robot. As specific actions, the server sends voice and action instructions to the robot to support the progress of the meeting.

[1551] Step 7:

[1552] The user follows instructions from the robot to conduct the meeting. The input is instructions from the robot, and the output is the progress of the meeting. In concrete terms, the user speaks and acts according to the robot's instructions to ensure the meeting proceeds smoothly.

[1553] Example 2

[1554] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1555] In conventional video conferencing systems, meeting facilitation is done manually, which places a heavy burden on attendees and can reduce the quality and effectiveness of meetings. Furthermore, patent-related discussions require specialized knowledge, making it difficult to generate appropriate questions and proposals. Furthermore, non-face-to-face communication often does not proceed smoothly, causing meetings to stall.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1557] In this invention, the server includes a means for automatically facilitating video conferences using artificial intelligence, a means for analyzing participants' remarks, facial expressions, and tone of voice to quantify their emotions, a means for optimally facilitating the conference, a means for collecting patent-related data, a means for preprocessing the collected data, a means for generating questions and suggestions using a generative AI model, a means for transmitting the generated questions and suggestions to a terminal, and a means for the terminal to present the results to the user. This reduces the burden on users and improves the quality and effectiveness of conferences. It also promotes specialized discussions related to patents and enables smooth communication even when not face-to-face.

[1558] A "video conference" is a type of conference in which multiple participants communicate in real time using video and audio via the Internet.

[1559] "Facilitation" is the activity of providing support and coordination to ensure the smooth progress of meetings and discussions.

[1560] "Artificial intelligence" is the technology that enables computer systems to learn, reason, and self-correct by imitating human intelligence.

[1561] "Speech content" refers to information and opinions verbally expressed by participants during a meeting or discussion.

[1562] "Facial expressions" are emotions and reactions shown by the movement of facial muscles.

[1563] "Vocal timbre" refers to the characteristics of a voice, such as its quality, tone, and pitch.

[1564] "Quantifying emotions" means expressing emotional states as numerical data.

[1565] A "patent" is a legal right that grants an inventor exclusive use of an invention for a certain period of time.

[1566] "Collecting data" is the act of gathering the necessary information.

[1567] "Preprocessing" is the initial process used to prepare data for easy analysis and use.

[1568] A "generative AI model" is a model that uses artificial intelligence to generate new data and information.

[1569] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific topic.

[1570] A "terminal" is a device that allows a user to input and output information.

[1571] A "user" is someone who uses a system or service.

[1572] "Presenting the results" means showing the generated information or data to the user.

[1573] This invention is a system that uses artificial intelligence to automatically facilitate video conferences. The system has the ability to analyze participants' remarks, facial expressions, and tone of voice to quantify their emotions. It also includes the ability to collect patent-related data, perform preprocessing, and generate questions and suggestions using a generative AI model. A specific embodiment of this system is described below.

[1574] Hardware and Software Configuration

[1575] server

[1576] The server comprises hardware and software to perform the following functions:

[1577] Data collection: The server collects the necessary information from patent-related databases (e.g., patent information databases).

[1578] Data preprocessing: The server uses Python's NLTK library and spaCy to preprocess the collected data.

[1579] Applying generative AI models: The server uses a pre-trained generative AI model (e.g., GPT-4) to generate patent-related questions and suggestions.

[1580] Sending results: The server sends the generated questions and suggestions to the device.

[1581] Terminal

[1582] The terminal is a device where the user can input information and display the results sent from the server. The terminal has the following functions:

[1583] Receiving user input: The terminal receives prompts entered by the user.

[1584] Displaying results: The device presents the user with the questions and suggestions sent by the server.

[1585] User

[1586] A user is someone who uses the system to hold a video conference. The user performs the following operations:

[1587] Entering a prompt: The user enters a prompt into the terminal to facilitate discussion related to the patent.

[1588] Specific examples

[1589] For example, consider a case where a user uses this system when holding a meeting regarding a patent.

[1590] Example prompt sentence:

[1591] "I'd like to discuss a new patent application. Please generate questions and suggestions regarding the technical details of the patent and the competitive situation in the market."

[1592] When a user enters this prompt into a terminal, the server collects and preprocesses relevant data from a patent information database. It then uses a generative AI model to generate appropriate questions and suggestions, which it then sends to the terminal. The terminal then presents the generated questions and suggestions to the user. For example, a question such as "How is this technology different from existing patents?" is displayed.

[1593] In this way, users can use the system to efficiently advance patent-related discussions. The system reduces the burden on users and improves the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[1594] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1595] Step 1:

[1596] The user enters a prompt statement.

[1597] The user inputs a prompt sentence into the terminal to promote discussions related to patents. For example, the user inputs, "I would like to hold a discussion about a new patent application. Please generate questions or suggestions about the technical details of the patent and the competitive situation in the market." The input prompt sentence is sent from the terminal to the server.

[1598] Step 2:

[1599] The server collects the data.

[1600] The server receives the prompt sent by the user and collects data related to patents. Specifically, it retrieves relevant patent information from a patent information database. For example, it calls the API of a patent information database to collect patent information related to "new technology." The input is the prompt, and the output is the collected patent information.

[1601] Step 3:

[1602] The server preprocesses the data.

[1603] The server preprocesses the collected patent information. It uses Python's NLTK library and spaCy to analyze the text data and extract important keywords and phrases. For example, it extracts keywords such as "technical details" and "market competitive situation." The input is the collected patent information, and the output is the preprocessed data.

[1604] Step 4:

[1605] The server applies the generative AI model.

[1606] The server generates questions and suggestions using a generative AI model (e.g., GPT-4) based on the preprocessed data. The generative AI model generates appropriate questions and suggestions based on the extracted keywords. For example, it generates questions such as, "How is this technology different from existing patents?" The input is the preprocessed data, and the output is the generated questions and suggestions.

[1607] Step 5:

[1608] The server transmits the generated results to the terminal.

[1609] The server sends the generated questions and suggestions to the terminal. The generated questions and suggestions are sent to the terminal in JSON format. For example, they are sent in the format {"question": "How is this technology different from existing patents?"}. The input is the generated questions and suggestions, and the output is the data sent to the terminal.

[1610] Step 6:

[1611] The terminal presents the results to the user.

[1612] The terminal presents the questions and suggestions received from the server to the user. The user can then advance the discussion based on the presented questions and suggestions. For example, the terminal screen might display "How is this technology different from existing patents?" The input is the data sent from the server, and the output is the questions and suggestions presented to the user.

[1613] (Application example 2)

[1614] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1615] In conventional video conferencing systems, meeting facilitation is done manually, which places a heavy burden on attendees and can reduce the quality and effectiveness of meetings. Furthermore, patent-related discussions and technical debates require specialized knowledge, making it difficult to ask appropriate questions or make appropriate suggestions. Furthermore, the inability to smoothly communicate without face-to-face contact is also an issue.

[1616] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1617] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing attendees' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for generating questions and suggestions to promote patent-related discussions, and means for generating prompt sentences based on the conference topic using a generative AI model.This improves the quality and effectiveness of conferences, efficiently supports patent-related discussions and technical debates, and enables smooth communication even when not face-to-face.

[1618] A "video conference" is a conference in which multiple participants share video and audio in real time via the Internet.

[1619] "Facilitation" is the activity of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[1620] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[1621] "Speech content" refers to information or opinions expressed orally by participants in a meeting or discussion.

[1622] "Facial expressions" refer to emotions and reactions expressed through the movement of facial muscles.

[1623] "Vocal timbre" refers to characteristics such as the quality of a voice, tone, and intonation.

[1624] "Quantifying emotions" means expressing emotional states as numerical data.

[1625] A "patent" is an exclusive right granted to an invention.

[1626] "Facilitating discussion" means supporting participants to actively exchange opinions and engage in lively debate.

[1627] "Generating questions and suggestions" refers to automatically creating appropriate questions and suggestions based on a specific topic.

[1628] A "generative AI model" is an artificial intelligence model that generates new data or information based on input data.

[1629] A "prompt" is an instruction or question that is input into a generative AI model.

[1630] "Inside a factory" refers to the inside of a facility where production activities in the manufacturing industry take place.

[1631] A "technical discussion" is an exchange of opinions and discussion between experts on technical matters.

[1632] A system for carrying out the present invention includes a program for automatically facilitating a video conference. The system is realized using the following hardware and software.

[1633] Hardware

[1634] Server: A server with high-performance computing power is required to run generative AI models and analyze data quickly.

[1635] Device: You will need a device (computer, tablet, smartphone, etc.) to participate in the video conference.

[1636] Camera and microphone: A camera and microphone are required to capture what participants are saying, their facial expressions, and tone of voice.

[1637] software

[1638] Generative AI model: Uses OpenAI's API to generate prompts based on the meeting topic.

[1639] Data analysis software: Software is needed to analyze what attendees say, their facial expressions, and tone of voice, and to quantify their emotions.

[1640] Video conferencing software: Software for conducting video conferences (e.g., Zoom or Microsoft Teams) is required.

[1641] Data processing and calculation

[1642] The server analyzes participants' comments, facial expressions, and tone of voice in real time during video conferences to quantify their emotions. This allows the system to grasp the progress of the conference and participants' reactions. Furthermore, it uses a generative AI model to generate prompts based on the conference topic, automatically asking questions and making suggestions to promote patent-related discussions.

[1643] Specific examples

[1644] For example, if a meeting is being held regarding the patenting of a new manufacturing process, the server might generate a prompt like this:

[1645] "What technical features should be highlighted to patent this manufacturing process?"

[1646] "What are the advantages of this process compared to competitor patents?"

[1647] This allows attendees to efficiently advance discussions, improving the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[1648] In this manner, the system for implementing the invention can automate the facilitation of video conferences to support patent-related discussions and technical debates.

[1649] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1650] Step 1:

[1651] At the start of a video conference, the server activates the camera and microphone to capture what participants are saying, their facial expressions, and tone of voice. As input, it receives real-time data from the camera and microphone, and as output, it sends this data to software for analysis.

[1652] Step 2:

[1653] The server uses analysis software to analyze the acquired data on speech content, facial expressions, and tone of voice, and quantifies emotions. It receives real-time data from the camera and microphone as input, and generates quantified emotional data as output. Specifically, it converts speech content into text using voice recognition technology, and analyzes facial muscle movements using facial expression recognition technology.

[1654] Step 3:

[1655] Based on the analysis results, the server uses a generative AI model to generate a prompt based on the meeting topic. It receives quantified emotion data and the meeting topic as input, and generates a prompt as output. Specifically, it generates a prompt using OpenAI's API.

[1656] Step 4:

[1657] The server sends the generated prompt sentence to the video conferencing software and presents it to the attendees. The server receives the generated prompt sentence as input and displays the prompt sentence on the video conferencing software as output. As a specific operation, the server displays the prompt sentence using a chat function or a screen sharing function of the video conferencing software.

[1658] Step 5:

[1659] The user advances the discussion based on the presented prompt sentence. The system receives the prompt sentence as input and generates the content of the discussion as output. As a specific action, the user expresses their opinion or question in response to the prompt sentence.

[1660] Step 6:

[1661] The server monitors the progress of the discussion and generates additional prompts as needed. It receives the content and emotion data of the discussion as input and generates additional prompts as output. Specifically, it uses the generative AI model again to generate new prompts.

[1662] Step 7:

[1663] At the end of the meeting, the server saves all data and generates a report of the meeting. It receives as input all data acquired during the meeting and generates as output a report of the meeting. Specific operations include saving the data in a database and creating a report using report generation software.

[1664] Example 3

[1665] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1666] With conventional video conferencing systems, it was difficult to analyze attendees' emotions in real time and adjust the progress of the meeting appropriately. This resulted in problems such as a decline in the quality and effectiveness of the meeting and an increase in the burden on users. In addition, non-face-to-face communication was often not possible, which caused the progress of the meeting to be delayed.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for automatically faciliting the conference using artificial intelligence, means for analyzing the content of attendees' remarks, facial expressions, and tone of voice and quantifying their emotions, means for optimally facilitating the conference, means for collecting audio data of the conference, means for transmitting the collected audio data to the server, means for the server to perform emotion analysis of the audio data, means for adjusting the progress of the conference based on the analysis results, and means for suggesting appropriate actions to the user. This improves the quality and effectiveness of the conference, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1668] "Means for automated meeting facilitation using artificial intelligence" refers to a function that uses artificial intelligence to automatically progress and coordinate meetings.

[1669] "Means of analyzing attendees' comments, facial expressions, and tone of voice and quantifying their emotions" is a function that analyzes attendees' comments, facial expressions, and tone of voice and expresses their emotions as a number.

[1670] "Means for optimal meeting facilitation" is a function that makes adjustments to optimize the progress of meetings based on analyzed emotional data.

[1671] The "means for collecting conference audio data" is a function for collecting audio during a conference in real time.

[1672] The "means for transmitting collected voice data to a server" is a function for transmitting collected voice data to a server via the Internet.

[1673] The "means for the server to analyze the emotions of the voice data" is a function for the server to analyze the voice data received and identify the emotions of the attendees.

[1674] "Means for adjusting the progress of the meeting based on the analysis results" is a function for appropriately adjusting the progress of the meeting based on the results of emotion analysis.

[1675] The "means for suggesting appropriate actions to the user" is a function for suggesting appropriate actions to the user based on the analysis results.

[1676] This invention relates to a system that uses artificial intelligence to automatically facilitate meetings. This system analyzes the content, facial expressions, and tone of voice of attendees and quantifies their emotions to enable optimal meeting facilitation.

[1677] 1. Generate the system program

[1678] The system was developed using Python, with the main libraries being "Hugging Face Transformers" for emotion analysis and "Google Calendar API" for meeting management.

[1679] 2. Explain the program's processing in natural language

[1680] When a user starts a meeting, the device collects audio data in real time. The collected audio data is sent to a server, where it undergoes emotion analysis using Hugging Face Transformers. The analysis results are classified into three categories: positive, negative, and neutral.

[1681] The server adjusts the progress of the meeting based on the analysis results. For example, if it determines that attendees' emotions are leaning negatively, the server will pause the meeting through the Google Calendar API and display a message on the device suggesting a break.

[1682] 3. Add examples to your description

[1683] As a concrete example, the following scenario can be considered.

[1684] scenario:

[1685] A user starts an online meeting. During the meeting, one of the attendees repeatedly makes negative comments. The system performs sentiment analysis on the comments and determines that the comments have a strong negative sentiment. The server pauses the meeting and displays a message on the device asking, "Suggest taking a break?"

[1686] Example prompt sentence:

[1687] "Develop a system that analyzes attendees' emotions in real time during online meetings and suggests pausing the meeting and taking a break if negative emotions are present."

[1688] This system provides an environment in which smooth communication can be achieved even without face-to-face contact. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1689] Step 1:

[1690] A user starts a conference. The user launches the conference application and clicks the "Start Conference" button. This causes the system to start collecting voice data. The input is the user's operation, and the output is the conference start signal.

[1691] Step 2:

[1692] The device collects the audio data of the meeting. The device picks up the audio during the meeting with a microphone and converts it into digital audio data in real time. For example, when a user says "Hello, everyone," the audio is immediately collected as digital data. The input is the audio during the meeting, and the output is digital audio data.

[1693] Step 3:

[1694] The terminal sends the collected voice data to the server. The terminal sends the collected voice data to the server using the HTTPS protocol. The data is AES encrypted and transferred securely. The input is digital voice data, and the output is a transmission completion signal to the server.

[1695] Step 4:

[1696] The server performs emotion analysis on the voice data. The server inputs the received voice data into the Hugging Face Transformers emotion analysis model. For example, a statement such as "This project might not work out" is analyzed as negative. The input is digital voice data, and the output is the emotion analysis result.

[1697] Step 5:

[1698] The server adjusts the progress of the meeting based on the analysis results. If the server determines that the result of the sentiment analysis is negative, it issues an instruction to pause the progress of the meeting using the Google Calendar API. The input is the sentiment analysis result, and the output is an instruction to adjust the progress of the meeting.

[1699] Step 6:

[1700] The terminal suggests appropriate actions to the user. The terminal receives instructions from the server and displays a message to the user saying, "Attendees' emotions are negative. Would you like to suggest taking a break?" The user adjusts the progress of the meeting by selecting "Yes" or "No." The input is an instruction from the server, and the output is a suggested message to the user.

[1701] In this way, the system provides an environment in which smooth communication can be carried out even without face-to-face contact.

[1702] (Application example 3)

[1703] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1704] In conventional video conferencing systems and work environments, it was difficult to properly analyze the emotions of attendees and workers and optimize meeting progress and work efficiency. Furthermore, when emotions turned negative, it was not possible to respond appropriately, which led to problems with reduced meeting quality and work safety. This increased the burden on users and hindered smooth non-face-to-face communication.

[1705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1706] In this invention, the server includes means for automatically facilitating video conferences using AI, means for analyzing the content of participants' remarks, facial expressions, and tone of voice to quantify their emotions, means for optimally facilitating the conference, means for analyzing the emotions of workers in real time to improve work efficiency and safety, and means for pausing work and suggesting a break when emotions become negative.This optimizes the progress of the conference and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1707] A "video conference" is a conference in which multiple participants share video and audio via the Internet.

[1708] "Facilitation" means providing support and coordination to ensure that meetings and work progress smoothly.

[1709] "AI" stands for artificial intelligence, a technology that allows machines to imitate human intelligence and learn and reason.

[1710] "Speech" refers to information and opinions that participants verbally express during a meeting or task.

[1711] "Facial expression" refers to emotions and intentions shown by facial muscle movements.

[1712] "Tone" refers to the quality and tone of a voice, and reflects the speaker's emotions and state.

[1713] "Quantifying emotions" means expressing emotions as quantitative data.

[1714] A "worker" is someone who performs a specific task in a factory, office, etc.

[1715] "Real time" means dealing with what's happening right now.

[1716] "Work efficiency" refers to the ability to minimize waste of time and effort while performing work and achieve maximum results.

[1717] "Safety" means a state in which work or activities can be carried out without risk and with peace of mind.

[1718] "Suggesting a break" means temporarily pausing work or the progress of a meeting to encourage participants to take a rest.

[1719] A "system" is a mechanism in which multiple elements work together to perform a specific function.

[1720] A system for implementing this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content, facial expressions, and tone of voice of attendees and quantifying their emotions, means for optimally facilitating meetings, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for pausing work and suggesting a break when emotions become negative.

[1721] System Program

[1722] The program of this system performs the following processing.

[1723] 1. Hardware: The system uses a camera (e.g., Logitech C920) to capture the faces of attendees and workers.

[1724] 2. Software: The system uses software such as OpenCV, Keras, and TensorFlow to process the captured images and analyze emotions.

[1725] 3. Data processing: The captured image is converted to grayscale and face detection is performed. The detected face regions are preprocessed to be input into the emotion analysis model.

[1726] 4. Data computation: Predict emotions from preprocessed facial images using an emotion analysis model.

[1727] Specific examples of processing

[1728] For example, if a worker in a factory feels stressed, the system will analyze their emotions in real time and suggest pausing work and taking a break, improving work efficiency and safety while reducing the burden on the worker.

[1729] Prompt Sentence Examples

[1730] An example of a prompt to input to a generative AI model is as follows:

[1731] Develop a robot assistant application in a factory that analyzes workers' emotions in real time to improve work efficiency and safety. Include a function that allows the robot to pause work and suggest a break if the worker is feeling stressed.

[1732] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1733] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1734] Step 1:

[1735] The server uses a camera to capture the faces of attendees and workers. The input is video data from the camera, and the output is the captured image data. Specifically, the camera acquires video in real time and captures the video frame by frame.

[1736] Step 2:

[1737] The server converts the captured image data to grayscale. The input is the captured image data, and the output is grayscale image data. Specifically, it uses OpenCV to convert the image to grayscale.

[1738] Step 3:

[1739] The server detects faces from grayscale images. The input is grayscale image data, and the output is coordinate data of the detected face area. Specifically, the server identifies the face area using the OpenCV face detection algorithm.

[1740] Step 4:

[1741] The server preprocesses the detected face region to input it into the emotion analysis model. The input is the coordinate data and grayscale image data of the face region, and the output is the preprocessed face image data. Specifically, the server cuts out the face region, resizes it to 48x48 pixels, and normalizes it.

[1742] Step 5:

[1743] The server inputs the pre...

Claims

1. A method for artificial intelligence to automatically facilitate video conferences, means for converting said videoconference voice data into text using speech recognition technology; A means for analyzing facial expressions from video data of the video conference using image recognition technology; A means for analyzing the content of remarks, facial expressions, and tone of voice of attendees in the video conference and quantifying their emotions; A means for optimally facilitating a meeting based on the emotion data quantified by the quantification means; A means to provide feedback based on meeting progress and sentiment data; A system including:

2. The system of claim 1 , further comprising means for providing expert facilitation based on the analysis results.

3. 2. The system according to claim 1, further comprising: a means for reducing a burden on a user and improving the quality and effectiveness of a conference; and a means for providing an environment in which smooth communication can be carried out even in a non-face-to-face manner.

Images