system

Abstract

To provide a system.SOLUTION: A system includes: means for automatically facilitating a video conference by artificial intelligence; means for converting audio data of the video conference into text using voice recognition technology; means for analyzing facial expressions from video data on the video conference using an image recognition technology; means for analyzing remarks, facial expressions, and tone of voice of attendees at the video conference and digitizing their emotions; means for optimally facilitating the conference on the basis of emotional data digitized by the digitizing means; and means for providing feedback on the basis of the progress of the conference and the emotional data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the spread of remote work, in modern society where video conferences are increasing, there is a need to smooth non-face-to-face communication and improve the quality of meetings. However, existing video conferencing tools have a problem that it is difficult to accurately grasp the emotions of the participants and perform appropriate facilitation.

Means for Solving the Problems

[0005] This invention provides a system that automatically facilitates video conferences using AI. By analyzing the content of participants' statements, facial expressions, and tone of voice, and quantifying their emotions, it performs optimal conference facilitation. Furthermore, it offers specialized facilitation services, such as patent-focused or strategy-focused services, reducing the burden on users and improving the quality and effectiveness of meetings. This provides an environment where smooth communication is possible even without face-to-face interaction. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1 of Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 of Form Example 2. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2. [Figure 15] It is a sequence diagram showing the processing flow of the data processing system in Example 3 of Form Example 3. [Figure 16] It is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3. [Figure 17] It is a sequence diagram showing the processing flow of the data processing system in Example 1 of Form Example 1 when combined with an emotion engine. [Figure 18] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] It is a sequence diagram showing the processing flow of the data processing system in Example 2 of Form Example 2 when combined with an emotion engine. [Figure 20] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] It is a sequence diagram showing the processing flow of the data processing system in Example 3 of Form Example 3 when combined with an emotion engine. [Figure 22] It is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine.

Embodiments for Carrying Out the Invention

[0007] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the language used in the following description will be explained.

[0009] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)), etc.

[0010] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0011] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0012] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0014] [First Embodiment]

[0015] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0016] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0027] "Example of form 1"

[0028] One embodiment of the present invention is a system in which AI automatically facilitates video conferences. Specifically, it is equipped with an AI engine that analyzes the content of what attendees say, their facial expressions, and their tone of voice. This AI engine uses speech recognition technology and image recognition technology to quantify the emotions of the attendees and performs optimal conference facilitation based on the results.

[0029] "Example of form 2"

[0030] Furthermore, it also features functions that provide specialized facilitation, such as patent-focused or strategy-focused facilitation. For example, in patent-focused facilitation, AI automatically asks questions and makes suggestions to facilitate discussions related to patents. This reduces the burden on users and improves the quality and effectiveness of meetings.

[0031] "Example of form 3"

[0032] Furthermore, it also has a function that provides an environment for smooth communication even when not face-to-face. Specifically, the AI ​​analyzes the emotions of the attendees and adjusts the progress of the meeting based on the results. For example, if it determines that the attendees' emotions are leaning towards negative, the AI ​​can automatically perform appropriate facilitation, such as pausing the meeting and suggesting a break to the attendees, just as a human would.

[0033] The following describes the processing flow for each example of the form.

[0034] "Example of form 1"

[0035] Step 1: The AI ​​engine starts the video conference and analyzes the participants' statements, facial expressions, and tone of voice in real time.

[0036] Step 2: The AI ​​engine uses speech recognition and image recognition technologies to quantify the emotions of the attendees.

[0037] Step 3: The AI ​​engine performs optimal meeting facilitation based on the analysis results. For example, if it determines that the attendees' emotions are leaning towards negative, it will take action such as pausing the meeting and suggesting a break to the attendees.

[0038] "Example of form 2"

[0039] Step 1: The AI ​​engine provides specialized facilitation, such as patent-focused or strategy-focused services.

[0040] Step 2: In patent-focused facilitation, AI automatically asks questions and makes suggestions to facilitate discussions related to patents.

[0041] Step 3: This reduces the burden on users and improves the quality and effectiveness of meetings.

[0042] "Example of form 3"

[0043] Step 1: Provide an environment where the AI ​​engine can communicate smoothly even without face-to-face interaction.

[0044] Step 2: The AI ​​analyzes the emotions of the attendees and adjusts the meeting's progress based on the results.

[0045] Step 3: For example, if the AI ​​determines that the attendees' emotions are leaning towards the negative, it can automatically perform appropriate facilitation actions similar to those a human would, such as pausing the meeting and suggesting a break to the attendees.

[0046] (Example 1)

[0047] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0048] Traditional video conferencing systems struggled to properly analyze participants' speech, facial expressions, and tone of voice to optimize meeting progress. Furthermore, the lack of means to provide feedback to improve meeting quality and effectiveness resulted in a significant burden on users and hindered smooth non-face-to-face communication.

[0049] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0050] In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice and quantifying their emotions, means for converting audio data into text using speech recognition technology, means for analyzing facial expressions from video data using image recognition technology, means for performing optimal conference facilitation based on the quantified emotion data, and means for providing feedback based on the progress of the conference and the emotion data. This makes it possible to analyze the emotions of attendees in real time and achieve optimal conference progress. Furthermore, by providing feedback to improve the quality and effectiveness of the conference, the burden on users is reduced and smooth communication can be achieved even without face-to-face interaction.

[0051] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[0052] "Facilitation" refers to techniques used to facilitate meetings and group discussions, enabling participants to effectively exchange opinions.

[0053] "Artificial intelligence" refers to a computer system that mimics human intelligence and possesses functions such as learning, reasoning, and recognition.

[0054] "Statements" refer to the words and opinions that participants express orally during a meeting.

[0055] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[0056] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and intonation, and is an element that conveys the speaker's emotions and intentions.

[0057] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[0058] "Quantifying emotions" means quantitatively evaluating emotions and expressing them as numerical values.

[0059] "Speech recognition technology" is a technology that converts speech into text, allowing the speaker's words to be automatically recorded as text.

[0060] "Image recognition technology" is a technique that analyzes image data and recognizes specific patterns or features.

[0061] "Optimal meeting facilitation" refers to techniques for conducting a meeting most effectively, taking into account the circumstances and emotions of the participants.

[0062] "Feedback" refers to evaluations and advice provided based on the progress and results of a meeting.

[0063] "Progress status" refers to how the meeting is progressing, indicating its current progress and status.

[0064] "Emotional data" refers to data that quantifies the emotions of participants, and can be used to facilitate and manage meetings.

[0065] This invention relates to a system that uses artificial intelligence to automatically facilitate video conferences. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to achieve optimal conference facilitation.

[0066] Hardware and software to be used

[0067] Hardware: Camera, microphone, device (PC, tablet, smartphone)

[0068] Software: Speech recognition technology (e.g., Google® Speech-to-Text), image recognition technology (e.g., OpenCV), AI engine (e.g., TENSORFLOW®)

[0069] System operation

[0070] Subject: Server

[0071] The server provides a system that automatically facilitates video conferences. This system is equipped with an AI engine that analyzes the content of participants' speech, facial expressions, and tone of voice. The server uses speech recognition and image recognition technologies to quantify the emotions of participants. Based on this quantified emotion data, the server performs optimal meeting facilitation. It also provides feedback based on the progress of the meeting and the emotion data.

[0072] Subject: terminal

[0073] The device transmits video and audio from the video conference to the server. The device uses its camera and microphone to capture participants' facial expressions and speech. This data is sent to the server in real time and analyzed by an AI engine.

[0074] Subject: User

[0075] Users participate in video conferences and send their speech and facial expressions to the server via their devices. Users simply participate in the meeting naturally without performing any special operations. The content of the user's speech, facial expressions, and tone of voice are analyzed by an AI engine to provide optimal meeting facilitation.

[0076] Specific example

[0077] Example 1: Analysis of speech during a meeting

[0078] When User A says, "What is the progress of this project?", the server uses speech recognition technology to convert the statement into text. The AI ​​engine then analyzes this text and recognizes that User A is asking a question. The server then facilitates the discussion, prompting other attendees to provide appropriate answers.

[0079] Example 2: Facial expression analysis

[0080] The terminal's camera captures user B's facial expression and sends it to the server. The server uses image recognition technology to analyze user B's facial expression and detects that user B is confused. The server pauses the meeting and facilitates the discussion by asking user B questions or comments.

[0081] Example of a prompt

[0082] Please describe a system that analyzes participants' speech, facial expressions, and tone of voice during video conferences to provide optimal meeting facilitation. Please include specific hardware and software names.

[0083] In this way, the system's operation can be explained from the perspectives of the server, terminal, and user, and understanding can be deepened with concrete examples. This system makes it possible to analyze the emotions of attendees in real time and achieve optimal meeting management. Furthermore, by providing feedback to improve the quality and effectiveness of meetings, it reduces the burden on users and enables smooth communication even in remote settings.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1: Start the video conference

[0086] Subject: User

[0087] The user starts a video conference using their device. The user launches the meeting application, enters the meeting ID, and joins the meeting. For example, if a user joins a meeting using a meeting app, they enter the meeting ID and click the "Join" button.

[0088] Input: Meeting ID, User actions

[0089] Output: Start video conference, prepare video and audio capture.

[0090] Step 2: Capture

[0091] Subject: terminal

[0092] The device uses a camera and microphone to capture the user's video and audio. The camera captures the user's facial expressions in real time, and the microphone records what the user says. For example, the device's camera captures the user's smile, and the microphone records them saying "hello."

[0093] Input: User video and audio

[0094] Output: Captured video and audio data

[0095] Step 3: Data transmission

[0096] Subject: terminal

[0097] The device sends the captured video and audio data to the server. The data is encrypted and sent securely to the server. For example, the device sends the captured video and audio data to the server using the HTTPS protocol.

[0098] Input: Captured video and audio data

[0099] Output: Data sent to the server

[0100] Step 4: Data Analysis

[0101] Subject: Server

[0102] The server analyzes the received video and audio data. It uses speech recognition technology to convert the audio data into text and image recognition technology to analyze facial expressions from the video data. For example, the server uses speech recognition technology to convert the audio "hello" into text and image recognition technology to detect the user's smile.

[0103] Input: Video and audio data sent to the server

[0104] Output: Text data, facial expression analysis results

[0105] Step 5: Quantifying emotions

[0106] Subject: Server

[0107] The server quantifies the user's emotions based on the analysis results. The AI ​​engine analyzes changes in voice tone and facial expressions to generate an emotion score. For example, the server might quantify the user's smile as "joy" and assign an emotion score of 80 / 100.

[0108] Input: Text data, facial expression analysis results

[0109] Output: Sentiment score

[0110] Step 6: Facilitating

[0111] Subject: Server

[0112] The server uses quantified sentiment data to provide optimal meeting facilitation. For example, if a user appears confused, the server pauses the meeting and prompts the user to ask questions or offer opinions.

[0113] Input: Sentiment score

[0114] Output: Facilitation instructions

[0115] Step 7: Provide feedback

[0116] Subject: Server

[0117] The server provides feedback based on the progress of the meeting and sentiment data. For example, after the meeting ends, it sends attendees a report about changes in their sentiment and the frequency of their comments during the meeting.

[0118] Input: Meeting progress, sentiment data

[0119] Output: Feedback Report

[0120] By specifically describing each processing step and clearly indicating the inputs and outputs, the system's operation can be understood more clearly.

[0121] (Application Example 1)

[0122] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0123] Conventional video conferencing systems have struggled to analyze participants' speech, facial expressions, and tone of voice to quantify their emotions and facilitate meetings effectively. Furthermore, while efficient meetings and discussions within factories require efficient progress, there was a lack of automated methods to manage meetings while considering participants' emotions. This resulted in decreased meeting quality and effectiveness, and increased burden on users.

[0124] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0125] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for installation in a robot for streamlining meetings and discussions within a factory, and means for optimizing meeting progress while considering the emotions of attendees. This makes it possible to conduct efficient meetings and discussions within a factory while considering the emotions of attendees.

[0126] A "video conference" is a meeting conducted via the internet, where multiple participants share video and audio in real time.

[0127] "Facilitation" refers to the methods and techniques used to ensure that meetings and discussions proceed smoothly.

[0128] "AI" is an abbreviation for artificial intelligence, which is a technology that allows computers to learn and reason by mimicking human intelligence.

[0129] "Statements" refer to the information and opinions that participants express orally during meetings or discussions.

[0130] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[0131] "Voice quality" refers to the tone and texture of a voice, and is an element that reflects the speaker's emotions and intentions.

[0132] "Quantifying emotions" means analyzing the emotional state of participants and expressing it as numerical data.

[0133] "Optimal meeting facilitation" means taking into account the participants' comments and emotional states to conduct the meeting in the most effective way possible.

[0134] A "factory meeting" refers to a meeting held to discuss matters related to the operation and production of the factory.

[0135] A "meeting" is when several people gather to exchange opinions and share information regarding a specific purpose or issue.

[0136] A "robot" is a mechanical device that operates automatically according to a program and performs specific tasks.

[0137] "Installation" refers to bringing software or applications into a computer or machine and making them usable.

[0138] "Optimizing meeting progress" means making the necessary adjustments and improvements to ensure that meetings are conducted efficiently and effectively.

[0139] As an embodiment of this invention, we describe a "smart meeting robot" system for streamlining meetings and discussions within a factory. This system uses AI to automatically facilitate video conferences, analyzing the content of participants' statements, facial expressions, and tone of voice to quantify their emotions and perform optimal meeting facilitation.

[0140] System Configuration

[0141] 1. Hardware:

[0142] Camera: Used to capture attendees' facial expressions in real time.

[0143] Microphone: Used to collect the content of what attendees say.

[0144] Robot: A mechanical device that moves around within a factory and supports the progress of meetings.

[0145] 2. Software:

[0146] Speech recognition library: The speech_recognition library is used to recognize attendees' speech in real time.

[0147] Emotion analysis model: Using the transformers library's pipeline, the model analyzes the facial expressions and tone of voice of attendees and quantifies their emotions.

[0148] Image processing library: The cv2 library is used to acquire video from the camera in real time and perform facial expression analysis.

[0149] System operation

[0150] 1. Speech recognition:

[0151] The server converts the audio data acquired from the microphone into text data using the speech_recognition library.

[0152] The converted text data will be used as information to optimize the meeting's progress.

[0153] 2. Emotion analysis:

[0154] The server processes the video data acquired from the camera in real time using the cv2 library and analyzes the facial expressions of the attendees.

[0155] The analyzed facial expression data is then converted into numerical emotion using the transformers library's pipeline.

[0156] 3. Optimizing meeting facilitation:

[0157] The server executes logic to optimize the meeting's progress based on the acquired audio and emotion data.

[0158] For example, if an attendee says, "There's a problem with this part," the robot will respond, "A problem has occurred. I will suggest a solution."

[0159] Additionally, if an attendee's facial expression is negative, the message "Attendees are feeling negative. We will adjust the meeting's progress." will be displayed.

[0160] Specific example

[0161] As a concrete example, let's consider a quality control meeting in a factory. If an attendee says, "There is a problem with this part," the robot will respond, "A problem has occurred. I will propose a solution." Also, if the attendee's expression is negative, it will display, "The attendee's mood is negative. I will adjust the meeting's progress."

[0162] Example of a prompt

[0163] Examples of prompt statements to input into a generative AI model include the following:

[0164] If someone says, "There's a problem with this part," please suggest how the meeting should proceed.

[0165] In this way, the smart meeting robot system can streamline meetings within factories and provide optimal facilitation that takes into account the emotions of the attendees.

[0166] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0167] Step 1:

[0168] The server acquires video data from the camera in real time. The input is video data from the camera, and the output is frame data processed using the image processing library cv2. Specifically, the server captures the video from the camera frame by frame, converts it to grayscale, and performs face detection.

[0169] Step 2:

[0170] The server acquires audio data from the microphone in real time. The input is audio data from the microphone, and the output is the spoken content converted into text data using the speech recognition library speech_recognition. Specifically, the server records the audio from the microphone and converts it to text using the speech recognition engine.

[0171] Step 3:

[0172] The server analyzes the facial expressions of attendees using the acquired frame data. The input is the frame data obtained in step 1, and the output is emotion data quantified using the pipeline of the emotion analysis model transformers. Specifically, the server extracts the region where a face was detected and inputs it into the emotion analysis model to quantify the emotion.

[0173] Step 4:

[0174] The server analyzes the acquired text data and extracts information necessary for the meeting's progress. The input is the text data obtained in step 2, and the output is instructions and suggestions for the meeting's progress. Specifically, the server inputs the text data into a natural language processing engine and extracts important keywords and phrases.

[0175] Step 5:

[0176] The server executes logic to optimize the meeting's progress based on sentiment data and text data. The inputs are the sentiment data obtained in step 3 and the text data obtained in step 4, and the output is specific actions for managing the meeting. Specifically, the server integrates the sentiment data and text data and generates instructions to adjust the meeting's progress.

[0177] Step 6:

[0178] The server sends the generated instructions to the robot to support the progress of the meeting. The input is the meeting progress instructions obtained in step 5, and the output is the specific actions taken by the robot. Specifically, the server sends voice and motion instructions to the robot to support the progress of the meeting.

[0179] Step 7:

[0180] The user conducts the meeting by following instructions from the robot. The input is the instructions from the robot, and the output is the progress of the meeting. Specifically, the user speaks and acts according to the robot's instructions to ensure the meeting runs smoothly.

[0181] (Example 2)

[0182] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0183] Traditional video conferencing systems often require manual facilitation, placing a heavy burden on attendees and potentially reducing the quality and effectiveness of meetings. Furthermore, discussions related to patents require specialized knowledge, making it difficult to generate appropriate questions and suggestions. Additionally, non-face-to-face communication is often inefficient, leading to delays in meeting progress.

[0184] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0185] In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice and quantifying their emotions, means for optimal conference facilitation, means for collecting patent-related data, means for pre-processing the collected data, means for generating questions and suggestions using a generative AI model, means for transmitting the generated questions and suggestions to a terminal, and means for the terminal to present the results to the user. This reduces the burden on the user and improves the quality and effectiveness of meetings. It also facilitates specialized discussions related to patents and enables smooth communication even in non-face-to-face settings.

[0186] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[0187] "Facilitation" refers to activities that provide support and coordination to ensure the smooth progress of meetings and discussions.

[0188] "Artificial intelligence" is a technology in which computer systems imitate human intelligence to learn, reason, and self-correct.

[0189] "Statements" refer to the information and opinions that participants express orally during a meeting or discussion.

[0190] "Facial expression" refers to emotions and reactions expressed through the movement of facial muscles.

[0191] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and pitch.

[0192] "Quantifying emotions" means expressing emotional states as numerical data.

[0193] A "patent" is a legal right that grants an inventor the exclusive right to use their invention for a certain period of time.

[0194] "Collecting data" is the act of gathering necessary information.

[0195] "Preprocessing" refers to the initial processing required to prepare data into a format that is easy to analyze and use.

[0196] A "generative AI model" is a model that uses artificial intelligence to generate new data and information.

[0197] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific theme.

[0198] A "terminal" is a device used by a user to input or output information.

[0199] A "user" is a person who uses a system or service.

[0200] "Presenting results" means showing the generated information or data to the user.

[0201] This invention is a system that uses artificial intelligence to automatically facilitate video conferences. The system has the function of analyzing the content of participants' speech, facial expressions, and tone of voice, and quantifying their emotions. It also includes a function to collect patent-related data, perform preprocessing, and generate questions and suggestions using a generative AI model. A specific embodiment of this system is described below.

[0202] Hardware and software configuration

[0203] server

[0204] The server is equipped with the hardware and software necessary to perform the following functions:

[0205] Data collection: The server collects the necessary information from patent-related databases (e.g., patent information databases).

[0206] Data preprocessing: The server uses Python's NLTK library and spaCy to preprocess the collected data.

[0207] Application of Generative AI Models: The server uses a pre-trained generative AI model (e.g., GPT-4®) to generate patent-related questions and suggestions.

[0208] Sending results: The server sends the generated questions and suggestions to the terminal.

[0209] terminal

[0210] A terminal is a device used by users to input information and display results sent from a server. A terminal has the following functions:

[0211] Receiving user input: The terminal receives prompts entered by the user.

[0212] Displaying results: The terminal presents the user with questions and suggestions sent from the server.

[0213] User

[0214] A user is someone who uses the system to conduct a video conference. The user performs the following actions:

[0215] Prompt input: The user enters prompts into the terminal to facilitate discussions related to the patent.

[0216] Specific example

[0217] For example, consider a scenario where a user is conducting a meeting about patents.

[0218] Example of a prompt:

[0219] "We would like to discuss a new patent application. Please generate questions and suggestions regarding the patent's technical details and the competitive landscape of the market."

[0220] When the user enters this prompt into the terminal, the server collects relevant data from the patent information database and performs preprocessing. Next, it uses a generative AI model to generate appropriate questions and suggestions and sends them to the terminal. The terminal then presents the generated questions and suggestions to the user. For example, a question such as "How does this technology differ from existing patents?" might be displayed.

[0221] In this way, users can use the system to efficiently advance discussions related to patents. The system can reduce the burden on users and improve the quality and effectiveness of meetings. It can also enable smooth communication even when meetings are not face-to-face.

[0222] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0223] Step 1:

[0224] The user enters a prompt message.

[0225] The user enters prompts into the terminal to facilitate discussions related to the patent. For example, they might enter, "I would like to discuss a new patent application. Please generate questions and suggestions regarding the patent's technical details and market competitive landscape." The entered prompts are then sent from the terminal to the server.

[0226] Step 2:

[0227] The server collects the data.

[0228] The server receives prompt messages from the user and collects patent-related data. Specifically, it retrieves relevant patent information from a patent information database. For example, it calls the API of the patent information database to collect patent information related to "new technologies." The input is the prompt message, and the output is the collected patent information.

[0229] Step 3:

[0230] The server preprocesses the data.

[0231] The server preprocesses the collected patent information. Using Python's NLTK library and spaCy, it analyzes the text data and extracts important keywords and phrases. For example, it extracts keywords such as "technical details" and "market competitive landscape." The input is the collected patent information, and the output is the preprocessed data.

[0232] Step 4:

[0233] The server applies the generated AI model.

[0234] The server generates questions and suggestions using a generative AI model (e.g., GPT-4) based on pre-processed data. The generative AI model generates appropriate questions and suggestions based on extracted keywords. For example, it might generate a question such as, "How does this technology differ from existing patents?" The input is pre-processed data, and the output is the generated questions and suggestions.

[0235] Step 5:

[0236] The server sends the generated results to the terminal.

[0237] The server sends the generated questions and suggestions to the terminal. The generated questions and suggestions are sent to the terminal in JSON format. For example, they are sent in the format {"question": "How does this technology differ from existing patents?"}. The input is the generated questions and suggestions, and the output is the data sent to the terminal.

[0238] Step 6:

[0239] The device presents the results to the user.

[0240] The terminal presents the user with questions and suggestions received from the server. The user can then proceed with the discussion based on the presented questions and suggestions. For example, the terminal screen might display "How does this technology differ from existing patents?". The input is data sent from the server, and the output is the questions and suggestions presented to the user.

[0241] (Application Example 2)

[0242] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0243] Traditional video conferencing systems require manual facilitation, which places a heavy burden on attendees and can lead to a decline in the quality and effectiveness of meetings. Furthermore, discussions related to patents and technical debates require specialized knowledge, making it difficult to ask appropriate questions or make suggestions. Additionally, the inability to facilitate smooth non-face-to-face communication was a problem.

[0244] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0245] In this invention, the server includes means for an AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for generating questions and suggestions to promote discussions related to patents, and means for generating prompt sentences based on the meeting topic using a generative AI model. This improves the quality and effectiveness of meetings, efficiently supports discussions and technical debates related to patents, and enables smooth communication even when not face-to-face.

[0246] A "video conference" is a meeting conducted via the internet, in which multiple participants share video and audio in real time.

[0247] "Facilitation" refers to activities that provide support and coordination to ensure that meetings and discussions proceed smoothly.

[0248] "AI" is an abbreviation for artificial intelligence, a technology in which computers imitate human intelligence to perform learning and reasoning.

[0249] "Statements" refer to the information and opinions expressed orally by participants during meetings or discussions.

[0250] "Facial expression" refers to emotions and reactions expressed through the movement of facial muscles.

[0251] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and intonation.

[0252] "Quantifying emotions" means expressing emotional states as numerical data.

[0253] A "patent" is an exclusive right granted to an invention.

[0254] "To facilitate discussion" means to support participants in actively exchanging opinions and engaging in lively debate.

[0255] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific topic.

[0256] A "generative AI model" is an artificial intelligence model that generates new data or information based on input data.

[0257] A "prompt statement" refers to an instruction or question that is input into a generative AI model.

[0258] "Factory interior" refers to the inside of a facility where production activities take place in the manufacturing industry.

[0259] A "technical discussion" is a meeting where experts exchange opinions and debate technical matters.

[0260] A system for carrying out this invention includes a program for automatically facilitating video conferences. The system is implemented using the following hardware and software.

[0261] hardware

[0262] Server: A server with high-performance computing capabilities is required. This will enable rapid execution of generative AI models and data analysis.

[0263] Device: You will need a device (such as a PC, tablet, or smartphone) to participate in the video conference.

[0264] Cameras and microphones: Cameras and microphones are necessary to capture what attendees say, their facial expressions, and their tone of voice.

[0265] software

[0266] Generative AI Model: Uses the OpenAI® API to generate prompt messages based on the meeting topic.

[0267] Data analysis software: Software is needed to analyze the content of attendees' statements, facial expressions, and tone of voice, and to quantify their emotions.

[0268] Video conferencing software: Software is required to conduct video conferences (for example, Zoom or MICROSOFT® TEAMS®).

[0269] Data processing and data calculation

[0270] The server analyzes the participants' statements, facial expressions, and tone of voice captured during video conferences in real time, quantifying their emotions. This allows for understanding the progress of the meeting and the participants' reactions. Furthermore, using a generative AI model, it generates prompt sentences based on the meeting topic and automatically asks questions and makes suggestions to facilitate discussions related to patents.

[0271] Specific example

[0272] For example, when conducting a meeting about a patent for a new manufacturing process, the server generates a prompt message like the following:

[0273] "What technical features should we emphasize to obtain a patent for this manufacturing process?"

[0274] “What is the superiority of this process compared to the patents of competing companies?”

[0275] This enables the attendees to conduct discussions efficiently, improving the quality and effectiveness of the meeting. Additionally, smooth communication is possible even in a non-face-to-face setting.

[0276] In this way, the system for implementing the invention can automate the facilitation of video conferences and support patent-related discussions and technical discussions.

[0277] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[0278] Step 1:

[0279] At the start of the video conference, the server activates the camera and microphone to acquire the speech content, expressions, and voices of the attendees. As input, it receives real-time data from the camera and microphone, and as output, it sends this data to software for analysis.

[0280] Step 2:

[0281] The server analyzes the acquired speech content, expression, and voice data using analysis software to quantify emotions. As input, it receives real-time data from the camera and microphone, and as output, it generates quantified emotion data. As specific operations, it uses speech recognition technology to convert the speech content into text and uses expression recognition technology to analyze the movements of facial muscles.

[0282] Step 3:

[0283] Based on the analysis results, the server uses the generated AI model to generate a prompt sentence based on the topic of the meeting. As input, it receives the quantified emotion data and the topic of the meeting, and as output, it generates a prompt sentence. As a specific operation, it uses the API of OpenAI to generate the prompt sentence.

[0284] Step 4:

[0285] The server sends the generated prompt message to the video conferencing software and presents it to the attendees. It receives the generated prompt message as input and displays the prompt message in the video conferencing software as output. Specifically, it displays the prompt message using the chat or screen sharing functions of the video conferencing software.

[0286] Step 5:

[0287] The user advances the discussion based on the presented prompt. It receives the prompt as input and generates the discussion content as output. Specifically, the user provides opinions and questions in response to the prompt.

[0288] Step 6:

[0289] The server monitors the progress of the discussion and generates additional prompts as needed. It receives discussion content and sentiment data as input and generates additional prompts as output. Specifically, it uses the generative AI model again to generate new prompts.

[0290] Step 7:

[0291] The server saves all data and generates a meeting report at the end of the meeting. It receives all data acquired during the meeting as input and generates a meeting report as output. Specifically, it saves the data to a database and creates the report using report generation software.

[0292] (Example 3)

[0293] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0294] Traditional video conferencing systems struggled to analyze participants' emotions in real time and appropriately adjust the meeting's progress. This resulted in decreased meeting quality and effectiveness, and increased user burden. Furthermore, the inability to communicate effectively without face-to-face interaction often led to delays in meetings.

[0295] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for artificial intelligence to automatically facilitate the meeting, means for analyzing the content of attendees' statements, facial expressions, and tone of voice and quantifying their emotions, means for optimal meeting facilitation, means for collecting meeting audio data, means for transmitting the collected audio data to the server, means for the server to perform emotion analysis on the audio data, means for adjusting the progress of the meeting based on the analysis results, and means for suggesting appropriate actions to the user. This improves the quality and effectiveness of meetings, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0296] "Methods for using artificial intelligence to automatically facilitate meetings" refers to functions that use artificial intelligence to automatically manage and coordinate meetings.

[0297] "A means of analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to a function that analyzes the statements, facial expressions, and tone of voice of attendees and expresses those emotions as numerical values.

[0298] "A means of optimally facilitating meetings" refers to a function that makes adjustments to optimize the progress of meetings based on analyzed emotional data.

[0299] "Means for collecting meeting audio data" refers to a function for collecting audio during a meeting in real time.

[0300] "Means for sending collected audio data to a server" refers to a function for sending collected audio data to a server via the internet.

[0301] The "means for the server to perform sentiment analysis on voice data" is a function for the server to analyze the received voice data and identify the emotions of the attendees.

[0302] The "means for adjusting the progress of the meeting based on the analysis results" is a function for appropriately adjusting the progress of the meeting based on the results of sentiment analysis.

[0303] The "means for proposing appropriate actions to the user" is a function for proposing appropriate actions to the user based on the analysis results.

[0304] This invention relates to a system in which artificial intelligence automatically facilitates meetings. This system can perform optimal meeting facilitation by analyzing the speech content, expressions, and voices of the attendees and quantifying their emotions.

[0305] 1. Generate the program of the system

[0306] The program of this system is developed using Python. As the main libraries, "Hugging Face Transformers" is used for sentiment analysis and "Google Calendar API" is used for meeting management.

[0307] 2. Explain the processing of the program in natural language

[0308] When the user starts a meeting, the terminal collects the voice data of the meeting in real time. The collected voice data is sent to the server, and sentiment analysis is performed using Hugging Face Transformers. The analysis results are classified into three categories: positive, negative, and neutral.

[0309] The server adjusts the meeting's progress based on the analysis results. For example, if it determines that an attendee's emotions are leaning towards negative, the server will pause the meeting via the Google Calendar API and display a message on the device suggesting a break.

[0310] 3. Add specific examples to the explanation.

[0311] The following scenario is a concrete example.

[0312] scenario:

[0313] A user starts an online meeting. During the meeting, one of the attendees repeatedly makes negative remarks. The system performs sentiment analysis on the remarks and determines that the negative emotions are strong. The server pauses the meeting and displays the message "Do you want to suggest taking a break?" on the user's device.

[0314] Example of a prompt:

[0315] "Develop a system that analyzes participants' emotions in real time during online meetings and suggests pausing the meeting and taking a break if negative emotions are strongly present."

[0316] This system provides an environment that enables smooth communication even without face-to-face interaction. The flow of a specific process in Example 3 will be explained using Figure 15.

[0317] Step 1:

[0318] The user initiates the meeting. The user launches the meeting application and clicks the "Start Meeting" button. This causes the system to begin collecting audio data. The input is the user's actions, and the output is the meeting start signal.

[0319] Step 2:

[0320] The device collects audio data from the meeting. The device uses a microphone to pick up audio during the meeting and converts it into digital audio data in real time. For example, if a user says "Hello everyone," that audio is immediately collected as digital data. The input is the audio from the meeting, and the output is digital audio data.

[0321] Step 3:

[0322] The terminal sends the collected audio data to the server. The terminal sends the collected audio data to the server using the HTTPS protocol. The data is AES encrypted and transmitted securely. The input is digital audio data, and the output is a signal indicating that transmission to the server is complete.

[0323] Step 4:

[0324] The server performs sentiment analysis on the audio data. The server inputs the received audio data into the Hugging Face Transformers sentiment analysis model. For example, the statement "This project might not work out" is analyzed as negative. The input is digital audio data, and the output is the sentiment analysis result.

[0325] Step 5:

[0326] The server adjusts the meeting's progress based on the analysis results. If the server determines that the sentiment analysis results are negative, it issues a command to temporarily suspend the meeting using the Google Calendar API. The input is the sentiment analysis result, and the output is a command to adjust the meeting's progress.

[0327] Step 6:

[0328] The terminal suggests appropriate actions to the user. The terminal receives instructions from the server and displays a message to the user saying, "Attendees are feeling negative. Would you like to suggest a break?" The user adjusts the meeting's progress by selecting "Yes" or "No." The input is the instruction from the server, and the output is the suggestion message to the user.

[0329] In this way, the system provides an environment where smooth communication can take place even without face-to-face interaction.

[0330] (Application Example 3)

[0331] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0332] Traditional video conferencing systems and work environments have struggled to adequately analyze the emotions of attendees and workers, thereby optimizing meeting progress and work efficiency. Furthermore, the inability to respond appropriately when emotions turned negative led to problems with reduced meeting quality and work safety. This resulted in increased user burden and hindered smooth non-face-to-face communication.

[0333] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means.

[0334] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for temporarily suspending work and suggesting a break if emotions become negative. This optimizes the progress of meetings and work efficiency, reduces the burden on users, and enables smooth communication even in non-face-to-face settings.

[0335] A "video conference" is a meeting conducted over the internet, where multiple participants share video and audio.

[0336] "Facilitation" refers to providing support and coordination to ensure that meetings and work proceed smoothly.

[0337] "AI" stands for artificial intelligence, which is a technology in which machines imitate human intelligence to learn and reason.

[0338] "Statements" refer to the information and opinions that participants express orally during meetings or work sessions.

[0339] "Facial expression" refers to the emotions and intentions conveyed through the movement of the facial muscles.

[0340] "Voice quality" refers to the sound quality and tone of a voice, and it reflects the speaker's emotions and state of mind.

[0341] "Quantifying emotions" means expressing emotions as quantitative data.

[0342] A "worker" is a person who performs a specific task in a factory, office, or similar setting.

[0343] "Real-time" means processing events that are currently unfolding immediately.

[0344] "Work efficiency" refers to the ability to minimize wasted time and effort during work and achieve maximum results.

[0345] "Safety" refers to a state in which work or activities can be carried out without danger and with peace of mind.

[0346] "Suggesting a break" means temporarily interrupting work or a meeting and encouraging participants to take a rest.

[0347] A "system" is a mechanism in which multiple elements work together to perform a specific function.

[0348] The system for implementing this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for temporarily suspending work and suggesting a break if emotions become negative.

[0349] System program

[0350] The program in this system performs the following operations:

[0351] 1. Hardware: The system uses a camera (e.g., Logitech C920) to capture the faces of attendees and workers.

[0352] 2. Software: The system uses software such as OpenCV, Keras, and TensorFlow to process captured images and analyze emotions.

[0353] 3. Data Processing: The captured images are converted to grayscale, and face detection is performed. The detected face regions are preprocessed for input into the emotion analysis model.

[0354] 4. Data processing: Use an emotion analysis model to predict emotions from pre-processed facial images.

[0355] Specific example of processing

[0356] For example, if a worker in a factory is experiencing stress, the system analyzes their emotions in real time, temporarily suspends their work, and suggests a break. This improves work efficiency and safety, and reduces the burden on workers.

[0357] Example of a prompt

[0358] Examples of prompts to input into a generative AI model are as follows:

[0359] Develop a robot assistant application that analyzes workers' emotions in real time within a factory to improve work efficiency and safety. Include a feature that allows the robot to pause work and suggest a break if a worker is experiencing stress.

[0360] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0361] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[0362] Step 1:

[0363] The server uses a camera to capture the faces of attendees and workers. The input is video data from the camera, and the output is the captured image data. Specifically, the camera acquires video in real time, and the server captures that video frame by frame.

[0364] Step 2:

[0365] The server converts the captured image data to grayscale. The input is the captured image data, and the output is grayscale image data. Specifically, it uses OpenCV to convert the image to grayscale.

[0366] Step 3:

[0367] The server detects faces from grayscale images. The input is grayscale image data, and the output is the coordinate data of the detected face region. Specifically, it uses the OpenCV face detection algorithm to identify the face region.

[0368] Step 4:

[0369] The server preprocesses the detected face regions for input into the emotion analysis model. The input consists of face region coordinate data and grayscale image data, and the output is the preprocessed face image data. Specifically, the server extracts the face region, resizes it to 48x48 pixels, and normalizes it.

[0370] Step 5:

[0371] The server inputs pre-processed facial image data into an emotion analysis model to predict emotions. The input is pre-processed facial image data, and the output is the predicted emotion. Specifically, Keras is used to input data into the emotion analysis model and obtain the prediction results.

[0372] Step 6:

[0373] The server determines whether the predicted emotion is negative. The input is the predicted emotion, and the output is the determination of whether it is a negative emotion. Specifically, it analyzes the prediction result and checks if it contains negative emotions (e.g., anger, sadness, fear).

[0374] Step 7:

[0375] The server pauses work and suggests a break if negative emotions are detected. The input is the result of the negative emotion detection, and the output is a notification suggesting a break. Specifically, the system pauses work and sends a notification to the user instructing them to take a break.

[0376] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0377] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0378] "Example of form 1"

[0379] In one embodiment of the present invention, an emotion engine recognizes the user's emotions and quantifies them. Specifically, it estimates emotions from the user's statements, tone of voice, facial expressions, etc., and quantifies them. This quantified emotion data is used to facilitate meetings. For example, if the AI ​​determines that the emotions of all participants are leaning towards negative, it takes actions such as pausing the meeting and suggesting a break.

[0380] "Example of form 2"

[0381] Furthermore, the emotion engine detects changes in users' emotions in real time and adjusts the meeting facilitation accordingly. For example, if the AI ​​senses that a participant is showing anger or dissatisfaction, it will take action such as providing that participant with an opportunity to speak or conveying that emotion to other participants. This ensures that the meeting proceeds smoothly and improves the overall satisfaction of the participants.

[0382] "Example of form 3"

[0383] Furthermore, the emotion engine predicts changes in users' emotions and optimizes meeting facilitation based on those predictions. For example, if it is predicted that a participant is likely to be dissatisfied with the meeting's progress, the AI ​​takes action such as providing that participant with an opportunity to speak in advance or adjusting the meeting's flow. This makes the meeting run more smoothly and further improves overall participant satisfaction.

[0384] The following describes the processing flow for each example of the form.

[0385] "Example of form 1"

[0386] Step 1: Estimate the user's emotions from their speech, tone of voice, facial expressions, etc.

[0387] Step 2: Quantify the estimated emotions.

[0388] Step 3: Utilize quantified emotional data to facilitate meetings.

[0389] Step 4: If the AI ​​determines that the overall sentiment of the participants is leaning towards negative, it will take action such as pausing the meeting and suggesting a break.

[0390] "Example of form 2"

[0391] Step 1: The emotion engine detects changes in the user's emotions in real time.

[0392] Step 2: Adjust the meeting facilitation according to changes in emotions.

[0393] Step 3: If the AI ​​senses that a participant is showing anger or dissatisfaction, it will take action such as giving that participant an opportunity to speak or conveying their feelings to other participants.

[0394] "Example of form 3"

[0395] Step 1: The emotion engine predicts changes in the user's emotions.

[0396] Step 2: Optimize meeting facilitation based on predictions.

[0397] Step 3: If the AI ​​predicts that a participant is likely to be dissatisfied with the meeting's progress, it will take action such as providing that participant with an opportunity to speak beforehand or adjusting the meeting's flow.

[0398] (Example 1)

[0399] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0400] Traditional video conferencing systems struggled to analyze participants' emotions and comments in real time and optimize meeting progress. Furthermore, there was a lack of means to provide professional facilitation to improve the quality and effectiveness of meetings. Additionally, there was no environment in place to reduce the burden on users and facilitate smooth communication in non-face-to-face settings.

[0401] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0402] In this invention, the server includes means for capturing the user's speech and facial expressions in real time using a high-resolution camera and microphone; means for transmitting the captured audio and video data to the server; means for converting the audio data into text using speech recognition software; means for analyzing the user's facial expressions from the video data using image recognition software; means for quantifying the user's emotions from the text and facial data using an emotion analysis engine; and means for optimizing the progress of the meeting based on the quantified emotion data. This makes it possible to analyze the emotions and speech of attendees in real time and provide optimal meeting facilitation.

[0403] A "high-resolution camera" is a camera that can capture high-resolution video to capture the user's face and facial expressions in detail.

[0404] A "microphone" is an audio input device used to accurately record what a user says.

[0405] "Audio data" refers to audio information, including the content of the user's speech, captured through a microphone.

[0406] "Video data" refers to video information, including the user's face and facial expressions, acquired through a high-resolution camera.

[0407] A "server" is a computer system used to receive and analyze audio and video data.

[0408] "Speech recognition software" is software that analyzes speech data and converts it into corresponding text data.

[0409] "Text data" refers to character information generated from speech data by speech recognition software.

[0410] "Image recognition software" is software that analyzes video data to recognize the user's facial expressions.

[0411] An "emotion analysis engine" is an analysis system that estimates and quantifies a user's emotions based on text data and facial expression data.

[0412] "Emotional data" refers to user emotional information quantified by an emotion analysis engine.

[0413] "Methods for optimizing meeting progress" refer to methods for adjusting the progress of a meeting and providing optimal facilitation based on quantified emotional data.

[0414] This invention relates to a system that uses AI to automatically facilitate video conferences. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to provide optimal conference facilitation.

[0415] Hardware and software to be used

[0416] Hardware: High-resolution camera, microphone, server

[0417] Software: Speech recognition software (e.g., Google Speech-to-Text), image recognition software (e.g., OpenCV), sentiment analysis engine (e.g., IBM Watson®)

[0418] System Overview

[0419] 1. Acquisition of user speech and facial expressions

[0420] The device uses a high-resolution camera and microphone to capture the user's speech and facial expressions in real time.

[0421] Specifically, the camera captures the user's face, and the microphone records the user's voice.

[0422] For example, if a user frowns and says, "This project is difficult," the camera and microphone capture that moment.

[0423] 2. Sending data

[0424] The device sends the captured audio and video data to the server.

[0425] Specifically, the device divides the data into packets and sends them to the server via the internet.

[0426] For example, user speech and facial expression data are sent to the server in real time.

[0427] 3. Speech Recognition and Image Recognition

[0428] The server uses speech recognition software to convert the audio data into text.

[0429] Simultaneously, the server uses image recognition software to analyze the user's facial expressions from the video data.

[0430] Specifically, speech recognition software analyzes the audio waveform and generates corresponding text. Image recognition software detects facial feature points and analyzes facial expressions.

[0431] For example, the statement "This project is difficult" is converted into text, and a frowning expression is analyzed as "confusion."

[0432] 4. Quantifying emotions

[0433] The server uses an emotion analysis engine to estimate the user's emotions from text data and facial expression data, and then quantifies them.

[0434] Specifically, the emotion analysis engine receives text and facial expression data as input and generates an emotion score.

[0435] For example, a confused expression and the statement "difficult" would result in a high negative emotion score.

[0436] 5. Meeting Facilitation

[0437] The server optimizes the meeting's progress based on quantified emotional data.

[0438] Specifically, the server analyzes the sentiment score, pauses the meeting as needed, and suggests a break.

[0439] For example, if the server determines that the overall sentiment score of the participants is leaning towards the negative, it will suggest, "Let's take a break."

[0440] Specific example

[0441] For example, the following scenario is possible.

[0442] Scenario: During the meeting, many participants looked tired, and their comments became increasingly negative.

[0443] process:

[0444] The device's camera and microphone capture the participants' facial expressions and speech.

[0445] The server uses speech recognition software to convert spoken content into text and image recognition software to analyze facial expressions.

[0446] The emotion analysis engine quantifies emotions from text and facial expression data and determines that there are many negative emotions.

[0447] The server pauses the meeting and suggests a break.

[0448] Example of a prompt

[0449] Examples of prompt statements to input into the generative AI model are as follows:

[0450] "If many participants in a meeting appear tired and their comments become increasingly negative, what are your suggestions for how to proceed with the meeting?"

[0451] By inputting this prompt into the AI ​​model, the AI ​​will suggest appropriate methods for facilitating meetings.

[0452] The flow of the specific processing in Example 1 will be explained using Figure 17.

[0453] Step 1:

[0454] The user speaks and shows facial expressions.

[0455] Input: User's statements and facial expressions.

[0456] Specific action: The user says, "This project is difficult," and frowns.

[0457] Step 2:

[0458] The device uses a high-resolution camera and microphone to capture the user's speech and facial expressions in real time.

[0459] Input: User's statements and facial expressions.

[0460] Output: Audio data and video data.

[0461] Specific operation: The camera captures the user's face, and the microphone records the user's voice.

[0462] Step 3:

[0463] The device sends the captured audio and video data to the server.

[0464] Input: Audio data and video data.

[0465] Output: Audio and video data sent to the server.

[0466] Specific operation: The terminal divides the data into packets and sends them to the server via the internet.

[0467] Step 4:

[0468] The server uses speech recognition software to convert the audio data into text.

[0469] Input: Audio data.

[0470] Output: Text data.

[0471] Specific operation: Speech recognition software analyzes the audio waveform and generates the corresponding text.

[0472] Step 5:

[0473] The server uses image recognition software to analyze the user's facial expressions from the video data.

[0474] Input: Video data.

[0475] Output: Facial expression data.

[0476] Specific operation: Image recognition software detects facial feature points and analyzes facial expressions.

[0477] Step 6:

[0478] The server uses an emotion analysis engine to quantify the user's emotions from text data and facial expression data.

[0479] Input: Text data and facial expression data.

[0480] Output: Sentiment data.

[0481] Specific operation: The emotion analysis engine receives text and facial expression data as input and generates an emotion score.

[0482] Step 7:

[0483] The server optimizes the meeting's progress based on quantified emotion data.

[0484] Input: Sentiment data.

[0485] Output: Instructions regarding the progress of the meeting.

[0486] Specific operation: The server analyzes the sentiment score, pauses the meeting as needed, and suggests a break.

[0487] (Application Example 1)

[0488] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0489] Conventional video conferencing systems and factory work support systems have struggled to adequately analyze the emotions of attendees and workers and provide optimal facilitation and work support based on that analysis. In particular, during long work sessions and meetings, stress and fatigue accumulate among participants and workers, leading to decreased efficiency and safety. Therefore, there is a need for new methods to improve the quality of meetings, work efficiency, and safety.

[0490] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0491] In this invention, the server includes means for an AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to support improvements in work efficiency and safety, means for suggesting breaks based on emotion recognition results, and means for notifying the suggestion by voice using a speech synthesis engine. This makes it possible to improve the efficiency and safety of meetings and work.

[0492] "Video conference facilitation" refers to the act of coordinating the progress of a video conference and encouraging participants to speak, thereby supporting the achievement of the meeting's objectives.

[0493] "Methods that AI automates" refers to methods or devices that use artificial intelligence to automatically perform specific tasks.

[0494] "Methods for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to methods or devices that use speech recognition technology or image recognition technology to analyze the content of attendees' statements, facial expressions, and tone of voice, and express the results as numerical values.

[0495] "Means for optimal meeting facilitation" refer to methods and devices for optimizing the progress of a meeting based on analyzed data.

[0496] "Means for analyzing workers' emotions in real time and providing support to improve work efficiency and safety" refers to methods and devices that analyze workers' emotions in real time and improve work efficiency and safety based on the results.

[0497] "Means for suggesting breaks based on emotion recognition results" refers to methods or devices for suggesting breaks to workers or attendees based on the results of emotion recognition.

[0498] "Means for notifying proposals by voice using a speech synthesis engine" refers to methods or devices for notifying proposal content by voice using speech synthesis technology.

[0499] The following system configuration is used as an embodiment of this invention. The server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to support improvements in work efficiency and safety, means for suggesting breaks based on emotion recognition results, and means for notifying the suggestion by voice using a speech synthesis engine.

[0500] Hardware and software to be used

[0501] Hardware: Webcam, microphone

[0502] Software: OpenCV, Keras, pyttsx3

[0503] Data processing and data calculation

[0504] The server acquires video and audio in real time using a webcam and microphone. The acquired video data is subjected to face recognition using OpenCV, and the facial region is extracted. The extracted facial region is input into a Keras emotion recognition model, and the emotion is quantified. The audio data is analyzed in the same way, and emotion is estimated from the tone of voice.

[0505] Based on the analyzed sentiment data, the server optimizes meeting facilitation. For example, if the overall sentiment of attendees is negative, it will pause the meeting and suggest a break. It will also suggest a break if a worker's sentiment indicates stress or fatigue. These suggestions are communicated via voice using a speech synthesis engine (pyttsx3).

[0506] Specific example

[0507] For example, consider a scenario where, in a factory, a worker is working for a long period of time, and an emotion recognition system detects the worker's stress level and suggests a break. In this case, the server uses a webcam and microphone to analyze the worker's facial expressions and tone of voice, and if it determines that the stress level is high, it notifies the worker via voice message saying, "Please take a break."

[0508] Examples of prompts to input into a generative AI model:

[0509] "Design a system that analyzes workers' emotions in real time within the factory and suggests breaks if stress or fatigue levels are high. Specifically, it would analyze workers' facial expressions and tone of voice, quantify their emotions, and make judgments based on that."

[0510] In this way, it becomes possible to improve the efficiency and safety of meetings and work.

[0511] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[0512] Step 1:

[0513] The server acquires video and audio in real time using a webcam and microphone. The input consists of video data from the webcam and audio data from the microphone. The output is the raw video and audio data acquired.

[0514] Step 2:

[0515] The server performs face recognition on the acquired video data using OpenCV. The input is the video data acquired in step 1. The output is image data with the face region extracted. Specifically, the server detects the position of the face from the video data and extracts that region.

[0516] Step 3:

[0517] The server inputs the extracted facial region into a Keras emotion recognition model to quantify the emotion. The input is the facial image data extracted in step 2. The output is numerical data indicating the emotion. Specifically, the facial image data is preprocessed and input into the emotion recognition model to predict the emotion.

[0518] Step 4:

[0519] The server analyzes the acquired audio data and estimates emotions from the tone of voice. The input is the audio data acquired in step 1. The output is numerical data indicating emotions based on the tone of voice. Specifically, the audio data is preprocessed, and audio features are extracted to estimate emotions.

[0520] Step 5:

[0521] The server integrates the emotion data obtained in steps 3 and 4 to perform an overall emotion assessment. The inputs are facial emotion data and vocal emotion data. The output is numerical data representing the overall emotion assessment. Specifically, it integrates the facial and vocal emotion data to calculate an overall emotion score.

[0522] Step 6:

[0523] The server provides optimal meeting facilitation and work support based on an overall sentiment assessment. The input is the overall sentiment assessment data obtained in step 5. The output is a specific action (e.g., suggesting a break). Specifically, if the sentiment assessment is leaning towards negative, the server will pause the meeting and suggest a break.

[0524] Step 7:

[0525] The server uses a speech synthesis engine (pyttsx3) to announce the proposal verbally. The input is the action determined in step 6. The output is the verbal notification. Specifically, it notifies attendees and workers verbally of the suggestion to take a break.

[0526] In this way, the server can analyze emotions in real time and provide optimal meeting facilitation and work support.

[0527] (Example 2)

[0528] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0529] Traditional video conferencing systems often require manual meeting management and facilitation, placing a significant burden on attendees and potentially reducing the quality and effectiveness of meetings. Furthermore, it was difficult to gauge attendees' emotions in real time and respond accordingly, hindering smooth communication. Additionally, generating appropriate questions and suggestions was challenging during specialized discussions, such as those focused on patents or strategies.

[0530] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice and quantifying their emotions, means for optimal meeting facilitation, means for generating relevant questions and suggestions based on the theme and purpose of the meeting, and means for adjusting the progress of the meeting based on emotion data. This makes it possible to reduce the burden on attendees and improve the quality and effectiveness of the meeting. In addition, real-time responses based on emotion data enable smooth communication. Furthermore, even in specialized discussions such as patent-focused or strategy-focused discussions, the depth and efficiency of the discussion can be improved by automatically generating appropriate questions and suggestions.

[0531] A "video conference" is a type of meeting in which multiple participants communicate in real time using audio and video over the internet.

[0532] "Facilitation" is the process of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[0533] Artificial intelligence is a computer system that imitates human intelligence and performs learning, reasoning, and problem-solving.

[0534] "Statements" refer to the information and opinions that participants express orally during a meeting.

[0535] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[0536] "Voice quality" refers to the characteristics of a voice, such as tone, pitch, and rhythm, and is an element that conveys the speaker's emotions and intentions.

[0537] "Quantifying emotions" means expressing analyzed emotional data as quantitative numerical values.

[0538] "Optimal meeting facilitation" means conducting and coordinating meetings in the most effective way, according to their purpose and circumstances.

[0539] The "meeting theme" refers to the main topics or issues that will be discussed at the meeting.

[0540] "Purpose" refers to the specific goals or outcomes that the meeting aims to achieve.

[0541] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on the meeting's theme and objectives.

[0542] "Emotional data" refers to the analysis results that show the emotional state of the participants.

[0543] "Adjusting the progress of a meeting" means managing the flow and progress of a meeting appropriately and making adjustments to ensure it runs smoothly.

[0544] A "patent-focused" conference is a conference format specifically designed for discussions and examinations related to patents.

[0545] A "strategy-focused" meeting is a conference format specifically designed for discussions and considerations related to strategy.

[0546] Modes for carrying out the invention

[0547] This invention is a system that uses artificial intelligence to automatically facilitate video conferences, and it has the function of analyzing the content of participants' speech, facial expressions, and tone of voice to quantify their emotions. It also includes a function to generate relevant questions and suggestions based on the theme and purpose of the meeting, and to adjust the progress of the meeting based on the emotion data.

[0548] System Configuration

[0549] This system consists of three main elements: servers, terminals, and users.

[0550] server

[0551] The server uses a generative AI model to generate questions and suggestions based on the meeting's theme and objectives. For example, OpenAI's GPT-4 is used as the generative AI model. The server receives input from the user, generates appropriate questions and suggestions based on that input, and sends them to the terminal. The server also receives sentiment data sent from the terminal and determines actions to adjust the meeting's progress.

[0552] terminal

[0553] The device uses a camera and microphone to detect the user's emotions in real time. For example, it uses Microsoft's Azure Cognitive Services Emotion API as its emotion engine. The device captures the user's facial expressions through the camera and records their voice tone through the microphone. This data is sent to the emotion engine, where the user's emotions are analyzed in real time. The device receives instructions from the server and adjusts the meeting's progress.

[0554] User

[0555] Users use a terminal to input the meeting topic and purpose. For example, they might input, "I want to discuss new patent ideas." If a user expresses anger or dissatisfaction during the meeting, the terminal sends that information to the server, which then uses that information to determine what actions to take to adjust the meeting's progress.

[0556] Specific example

[0557] As a concrete example, let's consider the case of patent-focused facilitation. If a user inputs "I want to discuss a new patent idea," the server uses a generative AI model to automatically generate questions such as "What is the technical novelty of this idea?" and "How does it differ from existing patents?"

[0558] As a concrete example of an emotion engine, consider a situation where a participant expresses dissatisfaction during a meeting. The terminal analyzes the participant's facial expressions and tone of voice to detect dissatisfaction. The server receives this information and instructs the terminal to provide an opportunity for the participant to speak, such as by asking, "Do you have any comments, Mr. / Ms. XX?"

[0559] Example of a prompt

[0560] "I'd like to discuss new patent ideas."

[0561] "Please generate questions about the novelty of this technology."

[0562] "Please suggest appropriate responses to participants who express dissatisfaction during the meeting."

[0563] In this way, the system can reduce the burden on users and improve the quality and effectiveness of meetings.

[0564] The flow of the specific processing in Example 2 will be explained using Figure 19.

[0565] Step 1:

[0566] The user enters the meeting topic and purpose. The user uses the terminal's keyboard to type, "I want to discuss new patent ideas." This input data is sent from the terminal to the server.

[0567] Step 2:

[0568] The server uses a generative AI model to generate questions and suggestions related to the meeting topic. The server inputs the topic and objectives received from the user into the generative AI model and generates questions such as "What is the technical novelty of this idea?" or "How does it differ from existing patents?" The generated questions are sent from the server to the terminal.

[0569] Step 3:

[0570] The device detects the user's emotions in real time. The device captures the user's facial expressions through the camera and records their voice tone through the microphone. This data is sent to an emotion engine, which analyzes the user's emotions in real time. For example, if the user is frowning, anger is detected. This emotion data is then sent from the device to a server.

[0571] Step 4:

[0572] The server receives sentiment data and determines actions to adjust the meeting's progress. For example, if a user expresses dissatisfaction, the server instructs the terminal to give that user an opportunity to speak. This instruction is sent from the server to the terminal.

[0573] Step 5:

[0574] The terminal follows instructions from the server and coordinates the progress of the meeting. The terminal receives instructions from the server and displays a message to the user saying, "Do you have any comments, [Username]?" When the user begins to speak, the terminal sends the content to the server, ensuring the meeting proceeds smoothly.

[0575] (Application Example 2)

[0576] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0577] Traditional video conferencing systems have struggled to detect participants' emotions in real time and adjust the meeting's progress accordingly. Furthermore, there has been a lack of specialized facilitation services tailored to specific patent or strategy areas, making it difficult to improve the quality and effectiveness of meetings. Additionally, there is a need for methods to appropriately manage participants' emotions and facilitate smooth communication in factory meetings and brainstorming sessions.

[0578] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0579] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for supporting meetings and brainstorming sessions within a factory, and means for detecting participants' emotions in real time using an emotion engine and adjusting the progress of the meeting. This makes it possible to detect participants' emotions in real time and adjust the progress of the meeting based on that, enabling the provision of specialized facilitation focused on patents or strategies. Furthermore, it makes it possible to appropriately manage participants' emotions and promote smooth communication in meetings and brainstorming sessions within a factory.

[0580] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[0581] "Facilitation" is the activity of facilitating the smooth progress of meetings and group discussions, and supporting all participants in effectively exchanging opinions.

[0582] "AI" is an abbreviation for artificial intelligence, a technology in which computers mimic human intelligence to perform functions such as learning, reasoning, and recognition.

[0583] "Statements" refer to the opinions and information that participants express orally during a meeting or discussion.

[0584] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[0585] "Voice quality" refers to the characteristics of a speaker's voice, such as its tone, rhythm, and other features, and is an element that conveys emotions and intentions.

[0586] "Quantifying emotions" means analyzing the emotional state of participants and expressing it as numerical data.

[0587] A "factory meeting" refers to a meeting held on the factory floor in a manufacturing company, where discussions take place regarding product production, quality control, and operational improvements.

[0588] A "brainstorming session" is a discussion format in which participants freely share ideas and find creative solutions.

[0589] An "emotion engine" is a technology that analyzes data such as participants' facial expressions and tone of voice to detect their emotional state in real time.

[0590] "Real-time detection" means analyzing data instantly and obtaining results without delay.

[0591] "To coordinate the progress of a meeting" means to manage the flow and progress of the meeting and intervene or make corrections as needed.

[0592] "Patent-focused facilitation" refers to specialized support activities aimed at facilitating discussions related to patents.

[0593] "Strategy-focused facilitation" refers to specialized support activities aimed at facilitating discussions on strategic matters within companies and organizations.

[0594] The system for carrying out this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for supporting meetings and brainstorming sessions within factories, and means for detecting participants' emotions in real time using an emotion engine and adjusting the progress of the meeting.

[0595] The server first captures video of meeting participants in real time using a camera (e.g., Logitech C920). Next, it uses an emotion detection engine called EmotionEngine to analyze the participants' emotions from the captured video and express them as numerical data. The analyzed emotion data is sent to a facilitation support AI called FacilitationAI, which proposes actions to optimize the progress of the meeting.

[0596] Specifically, if one participant expresses anger or dissatisfaction, FacilitationAI instructs the server to give that participant an opportunity to speak. Conversely, if a participant expresses joy or satisfaction, it suggests actions to further encourage their participation. This leads to smoother meeting progress and increased overall participant satisfaction.

[0597] For example, if a meeting is being held in a factory to discuss the introduction of a new product line, and one of the participants expresses dissatisfaction, the server can provide that participant with an opportunity to speak and convey their feelings to the other participants, thereby smoothing the progress of the meeting.

[0598] Examples of prompts for a generative AI model include the following:

[0599] "We are holding a meeting about introducing a new product line. One of the participants is expressing dissatisfaction. Please suggest what facilitation actions should be taken in this situation."

[0600] In this way, the server can detect participants' emotions in real time and adjust the meeting's progress accordingly. This enables the provision of specialized facilitation focused on patents or strategies, and allows for appropriate management of participants' emotions and smooth communication in meetings and brainstorming sessions within factories.

[0601] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[0602] Step 1:

[0603] The server captures video of meeting participants in real time using a camera (e.g., Logitech C920). The input is video data from the camera, and the output is the captured video frame. This video frame is used in subsequent processing steps.

[0604] Step 2:

[0605] The server uses EmotionEngine to analyze participants' facial expressions from captured video frames and quantify their emotions. The input is video frames, and the output is numerical data indicating each participant's emotional state. Specifically, EmotionEngine analyzes the video frames, detects facial feature points, and classifies the emotions.

[0606] Step 3:

[0607] The server sends the analyzed sentiment data to FacilitationAI, which then proposes actions to optimize the meeting's progress. The input is sentiment data, and the output is the proposed facilitation action. Specifically, FacilitationAI analyzes the sentiment data and determines the appropriate action (e.g., provide opportunities to speak, encourage participation).

[0608] Step 4:

[0609] The server executes the actions suggested by FacilitationAI. The input is the suggested action, and the output is the result of the executed action. Specifically, the server manages the flow of the meeting, provides participants with opportunities to speak as needed, and communicates emotional states to other participants.

[0610] Step 5:

[0611] The server monitors the progress of the meeting and, if necessary, uses EmotionEngine again to detect participants' emotions and send them to FacilitationAI. The input is the progress of the meeting and the latest video frame, and the output is updated emotion data and new facilitation actions. Specifically, the server periodically captures video frames and re-analyzes the emotions to optimize the meeting's progress.

[0612] Step 6:

[0613] The server saves all sentiment data and a history of facilitation actions after a meeting to help improve future meetings. The input is sentiment data and action history, and the output is the saved data. Specifically, the server saves the sentiment data and action history to a database, which is then used as reference material for the next meeting.

[0614] (Example 3)

[0615] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0616] Traditional video conferencing systems have made it difficult to grasp the emotional state of attendees in real time and appropriately adjust the meeting's progress. Furthermore, they were unable to predict changes in attendees' emotions and optimize the meeting's progress based on those predictions, which sometimes led to a decline in the quality and effectiveness of meetings. Additionally, non-face-to-face communication was often not smooth, increasing the burden on users.

[0617] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0618] In this invention, the server includes means for collecting attendee data, means for analyzing the collected data and evaluating the emotional state of the attendees, means for adjusting the progress of the meeting based on the evaluation results, means for predicting changes in the emotional state of the attendees and optimizing the progress of the meeting based on the predictions, and means for providing smooth communication even when not face-to-face. This makes it possible to grasp the emotional state of attendees in real time and automatically perform appropriate facilitation. Furthermore, by predicting changes in the emotional state of attendees and optimizing the progress of the meeting, the quality and effectiveness of the meeting can be improved. In addition, by providing smooth communication even when not face-to-face, the burden on users can be reduced.

[0619] "Means for collecting attendee data" refers to devices or software that collect data such as facial expressions and voices of attendees participating in a meeting in real time.

[0620] "Means for analyzing collected data and evaluating the emotional state of attendees" refers to algorithms or software that analyze collected facial and audio data to evaluate the emotional state of attendees (joy, anger, sadness, etc.).

[0621] "Means for adjusting the progress of a meeting based on evaluation results" refers to systems or processes for temporarily pausing a meeting or suggesting a break based on the results of an evaluation of emotional state.

[0622] "Methods for predicting changes in attendees' emotions and optimizing the progress of a meeting based on those predictions" refers to algorithms or software that predict changes in attendees' emotions based on past emotional data and adjust the progress of the meeting based on those predictions.

[0623] "Means of providing smooth communication even without face-to-face interaction" refers to systems and processes that enable attendees to communicate smoothly with each other even when they are not physically meeting face-to-face.

[0624] This invention is a system for facilitating non-face-to-face communication and includes an AI function for automatically adjusting the progress of meetings. Specific embodiments of this system are described below.

[0625] As soon as the meeting starts, the server collects facial expressions and audio data from attendees via cameras and microphones connected to their devices (PCs or smartphones). The devices then transmit this data to the server in real time. The hardware used includes high-resolution cameras, microphones, and the server. The software used includes emotion analysis engines (e.g., Microsoft Azure's Emotion API) and meeting management systems (e.g., Zoom or Microsoft Teams).

[0626] The server sends the collected facial and voice data to the emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the emotional state of the attendees (joy, anger, sadness, surprise, etc.) in real time. For example, if an attendee's facial expression becomes stern and their voice tone drops, the emotion analysis engine will determine that the attendee is experiencing negative emotions. Based on this information, the server pauses the meeting and suggests a break to the attendees.

[0627] Furthermore, the emotion engine predicts changes in attendees' emotions based on past data. For example, if it is predicted that an attendee is likely to be dissatisfied with the meeting's progress, the server will provide that attendee with an opportunity to speak beforehand. It will also take actions such as adjusting the meeting's progress.

[0628] The following scenarios are possible as specific examples:

[0629] Scenario: During the meeting, participant A begins to express dissatisfaction.

[0630] Processing: The emotion analysis engine analyzes participant A's facial expressions and tone of voice and detects dissatisfaction. The server pauses the meeting and gives participant A an opportunity to speak.

[0631] Examples of prompts to input into a generative AI model:

[0632] "Design a system that analyzes attendees' emotions in real time during a meeting and, if negative emotions are detected, pauses the meeting and suggests a break. Also, add a function to predict changes in attendees' emotions and optimize the meeting's progress."

[0633] In this way, the server analyzes the emotions of the attendees and automatically performs appropriate facilitation, thereby realizing an environment that provides smooth communication even in a non-face-to-face setting. The flow of the specific processing in Example 3 will be explained using Figure 21.

[0634] Step 1:

[0635] The server activates the camera and microphone connected to the terminal as soon as the meeting starts. The terminal captures the attendees' facial expressions and audio data in real time and sends it to the server. The input is the video and audio data from the camera and microphone, and the output is the raw data sent to the server.

[0636] Step 2:

[0637] The server sends the received facial and audio data to the emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the emotional state of the attendees. The input is the raw data sent from the server, and the output is the evaluation result of the emotional state (joy, anger, sadness, surprise, etc.). Specifically, the emotion analysis engine analyzes facial features (e.g., eyebrow movement, mouth shape) and voice tone.

[0638] Step 3:

[0639] The server receives the emotional state evaluation results returned from the emotion analysis engine and determines whether the attendees' emotions are positive or negative. The input is the evaluation results from the emotion analysis engine, and the output is the judgment result of the emotional state. Specifically, if the server determines, based on the evaluation results, that the attendees' emotions are negative, it will temporarily suspend the meeting.

[0640] Step 4:

[0641] The server instructs the terminal to display a message suggesting a break if negative emotions are detected. The input is the result of the emotional state assessment, and the output is the break suggestion message displayed on the terminal. Specifically, the server sends a message to the terminal such as, "The attendee's emotions are leaning towards negative, so we suggest a 5-minute break."

[0642] Step 5:

[0643] The server predicts changes in attendees' emotions based on past emotional data. The input is past emotional data, and the output is the predicted result of those changes. Specifically, the emotion engine uses a machine learning algorithm to predict whether a particular attendee is likely to be dissatisfied with the progress of the meeting.

[0644] Step 6:

[0645] The server takes actions to optimize the meeting's progress based on predictions of emotional changes. The input is the predicted emotional changes, and the output is the optimized meeting progress instructions. Specifically, the server provides attendees who may have predicted dissatisfaction with an opportunity to speak in advance, and also takes actions such as adjusting the meeting's progress.

[0646] In this way, the server analyzes the emotions of the attendees and automatically performs appropriate facilitation, creating an environment that provides smooth communication even when not face-to-face.

[0647] (Application Example 3)

[0648] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0649] Traditional video conferencing and customer service systems have struggled to accurately analyze the emotions of attendees and customers and adjust responses in real time based on that analysis. This has resulted in problems such as meetings not running smoothly, decreased participant satisfaction, and inadequate customer service leading to low customer satisfaction. There is a need to address these issues and improve the quality of meetings and customer service.

[0650] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means.

[0651] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing customers' facial expressions and tone of voice to determine their emotions in real time, means for proposing appropriate responses based on the emotion analysis results, and means installed on smart devices to support customer service. This makes it possible to conduct meetings smoothly, improve the overall satisfaction of participants, and enable prompt and appropriate customer service, thereby improving customer satisfaction.

[0652] "Video conference facilitation" refers to the coordination and management required to ensure the smooth running of a video conference.

[0653] "Means of AI automation" refers to methods or devices that enable artificial intelligence to automatically perform specific tasks.

[0654] "Methods for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to methods or devices for analyzing the statements, facial expressions, and tone of voice of meeting participants and expressing their emotions as numerical values.

[0655] "Means of optimal meeting facilitation" refers to methods and devices for optimizing the progress of a meeting.

[0656] "Means of analyzing a customer's facial expressions and tone of voice to determine their emotions in real time" refers to methods or devices for analyzing a customer's facial expressions and tone of voice to determine their emotions in real time.

[0657] "Means for proposing appropriate responses based on emotion analysis results" refers to methods or devices for proposing appropriate responses based on the results of emotion analysis.

[0658] "Means installed on smart devices to support customer service" refers to methods and devices installed on devices such as smartphones and smart glasses to support customer service.

[0659] This invention is a system that uses AI to analyze emotions during video conferences and customer service interactions, and proposes appropriate responses in real time. The system includes functions for automatically facilitating smooth video conference proceedings and functions to support customer service interactions.

[0660] System Configuration

[0661] This system consists of the following main components:

[0662] 1. Video conferencing facilitation module: AI automatically manages the progress of video conferences.

[0663] 2. Emotion Analysis Module: Analyzes the content of attendees' and customers' statements, facial expressions, and tone of voice to quantify their emotions.

[0664] 3. Response Proposal Module: Based on the emotion analysis results, it proposes the optimal meeting facilitation and customer response.

[0665] 4. Smart Device Interface: Installed on devices such as smartphones and smart glasses to support customer service.

[0666] Hardware and software to be used

[0667] Hardware: Smart glasses, smartphone, camera, microphone

[0668] Software: OpenCV (face detection), Keras (sentiment analysis model)

[0669] Data processing and data calculation

[0670] 1. Face Detection: Use smart glasses or smartphone cameras to capture the faces of customers and attendees in real time. OpenCV is used to detect faces and acquire facial expression data.

[0671] 2. Emotion Analysis: The acquired facial expression data is input into Keras's emotion analysis model to quantify emotions. This allows for real-time assessment of the emotional state of customers and attendees.

[0672] 3. Suggested Response: Based on the emotion analysis results, appropriate responses are suggested. For example, if a customer is dissatisfied, the smart glasses display will show "The customer is dissatisfied. Please improve your response."

[0673] Specific example

[0674] scenario

[0675] When a store employee interacts with a customer and the customer shows signs of dissatisfaction, the smart glasses display will show the message, "The customer is dissatisfied. Please improve your service." This allows the employee to quickly improve their service and increase customer satisfaction.

[0676] Example of a prompt

[0677] "Capture the customer's facial expressions and use an emotion analysis model to determine their emotions. If the customer is dissatisfied, display a message suggesting improvements to the service."

[0678] In this way, a system that supports video conferencing and customer service in physical stores can be realized. This system makes it possible to conduct meetings smoothly, improve the overall satisfaction of participants, and enable quick and appropriate customer service, thereby improving customer satisfaction.

[0679] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[0680] Step 1:

[0681] Face detection

[0682] The server captures the faces of customers and attendees in real time using smart glasses or smartphone cameras. It receives video data from the cameras as input and detects faces using OpenCV. The output includes the location information and facial expression data of the detected faces. Specifically, it analyzes the camera footage frame by frame to identify the contours of the faces.

[0683] Step 2:

[0684] Emotion analysis

[0685] The server inputs the facial expression data obtained in Step 1 into the Keras emotion analysis model. It receives facial position information and facial expression data as input, and uses the emotion analysis model to quantify emotions. The output is the emotional state of the customer or attendee (e.g., joy, sadness, anger). Specifically, it preprocesses the facial expression data and inputs it into the emotion analysis model to predict emotions.

[0686] Step 3:

[0687] Proposed solutions

[0688] The server proposes an appropriate response based on the emotion analysis results obtained in step 2. It receives the emotional state as input and determines the appropriate response using the response suggestion module. As output, it generates a message to be displayed on smart glasses or a smartphone screen. Specifically, it searches the database for a response corresponding to the emotional state and generates a response message.

[0689] Step 4:

[0690] Message display

[0691] The device displays the corresponding message generated in step 3 on the smart glasses or smartphone display. It receives the corresponding message as input and processes it for display. As output, it visually presents the message to the user. Specifically, it calls the display's API to display the message.

[0692] Step 5:

[0693] User response

[0694] Based on the message displayed in Step 4, the user takes appropriate action towards customers and attendees. The input is receiving the displayed message and implementing the appropriate response. The output is improving customer and attendee satisfaction. Specifically, the user adjusts customer interactions and meeting progress according to the message.

[0695] In this way, the system can analyze emotions in real time and suggest appropriate responses, thereby improving the quality of meetings and customer interactions.

[0696] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0697] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0698] Other examples of generative AI include Gemini® (registered trademark) (Internet search). <url: https: gemini.google.com ?hl="ja">) are some examples.

[0699] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0700] [Second Embodiment]

[0701] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0702] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0703] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0704] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0705] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0706] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0707] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0708] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0709] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0710] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0711] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0712] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0713] "Example of form 1"

[0714] One embodiment of the present invention is a system in which AI automatically facilitates video conferences. Specifically, it is equipped with an AI engine that analyzes the content of what attendees say, their facial expressions, and their tone of voice. This AI engine uses speech recognition technology and image recognition technology to quantify the emotions of the attendees and performs optimal conference facilitation based on the results.

[0715] "Example of form 2"

[0716] Furthermore, it also features functions that provide specialized facilitation, such as patent-focused or strategy-focused facilitation. For example, in patent-focused facilitation, AI automatically asks questions and makes suggestions to facilitate discussions related to patents. This reduces the burden on users and improves the quality and effectiveness of meetings.

[0717] "Example of form 3"

[0718] Furthermore, it also has a function that provides an environment for smooth communication even when not face-to-face. Specifically, the AI ​​analyzes the emotions of the attendees and adjusts the progress of the meeting based on the results. For example, if it determines that the attendees' emotions are leaning towards negative, the AI ​​can automatically perform appropriate facilitation, such as pausing the meeting and suggesting a break to the attendees, just as a human would.

[0719] The following describes the processing flow for each example of the form.

[0720] "Example of form 1"

[0721] Step 1: The AI ​​engine starts the video conference and analyzes the participants' statements, facial expressions, and tone of voice in real time.

[0722] Step 2: The AI ​​engine uses speech recognition and image recognition technologies to quantify the emotions of the attendees.

[0723] Step 3: The AI ​​engine performs optimal meeting facilitation based on the analysis results. For example, if it determines that the attendees' emotions are leaning towards negative, it will take action such as pausing the meeting and suggesting a break to the attendees.

[0724] "Example of form 2"

[0725] Step 1: The AI ​​engine provides specialized facilitation, such as patent-focused or strategy-focused services.

[0726] Step 2: In patent-focused facilitation, AI automatically asks questions and makes suggestions to facilitate discussions related to patents.

[0727] Step 3: This reduces the burden on users and improves the quality and effectiveness of meetings.

[0728] "Example of form 3"

[0729] Step 1: Provide an environment where the AI ​​engine can communicate smoothly even without face-to-face interaction.

[0730] Step 2: The AI ​​analyzes the emotions of the attendees and adjusts the meeting's progress based on the results.

[0731] Step 3: For example, if the AI ​​determines that the attendees' emotions are leaning towards the negative, it can automatically perform appropriate facilitation actions similar to those a human would, such as pausing the meeting and suggesting a break to the attendees.

[0732] (Example 1)

[0733] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0734] Traditional video conferencing systems struggled to properly analyze participants' speech, facial expressions, and tone of voice to optimize meeting progress. Furthermore, the lack of means to provide feedback to improve meeting quality and effectiveness resulted in a significant burden on users and hindered smooth non-face-to-face communication.

[0735] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0736] In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice and quantifying their emotions, means for converting audio data into text using speech recognition technology, means for analyzing facial expressions from video data using image recognition technology, means for performing optimal conference facilitation based on the quantified emotion data, and means for providing feedback based on the progress of the conference and the emotion data. This makes it possible to analyze the emotions of attendees in real time and achieve optimal conference progress. Furthermore, by providing feedback to improve the quality and effectiveness of the conference, the burden on users is reduced and smooth communication can be achieved even without face-to-face interaction.

[0737] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[0738] "Facilitation" refers to techniques used to facilitate meetings and group discussions, enabling participants to effectively exchange opinions.

[0739] "Artificial intelligence" refers to a computer system that mimics human intelligence and possesses functions such as learning, reasoning, and recognition.

[0740] "Statements" refer to the words and opinions that participants express orally during a meeting.

[0741] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[0742] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and intonation, and is an element that conveys the speaker's emotions and intentions.

[0743] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[0744] "Quantifying emotions" means quantitatively evaluating emotions and expressing them as numerical values.

[0745] "Speech recognition technology" is a technology that converts speech into text, allowing the speaker's words to be automatically recorded as text.

[0746] "Image recognition technology" is a technique that analyzes image data and recognizes specific patterns or features.

[0747] "Optimal meeting facilitation" refers to techniques for conducting a meeting most effectively, taking into account the circumstances and emotions of the participants.

[0748] "Feedback" refers to evaluations and advice provided based on the progress and results of a meeting.

[0749] "Progress status" refers to how the meeting is progressing, indicating its current progress and status.

[0750] "Emotional data" refers to data that quantifies the emotions of participants, and can be used to facilitate and manage meetings.

[0751] This invention relates to a system that uses artificial intelligence to automatically facilitate video conferences. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to achieve optimal conference facilitation.

[0752] Hardware and software to be used

[0753] Hardware: Camera, microphone, device (PC, tablet, smartphone)

[0754] Software: Speech recognition technology (e.g., Google Speech-to-Text), image recognition technology (e.g., OpenCV), AI engine (e.g., TensorFlow)

[0755] System operation

[0756] Subject: Server

[0757] The server provides a system that automatically facilitates video conferences. This system is equipped with an AI engine that analyzes the content of participants' speech, facial expressions, and tone of voice. The server uses speech recognition and image recognition technologies to quantify the emotions of participants. Based on this quantified emotion data, the server performs optimal meeting facilitation. It also provides feedback based on the progress of the meeting and the emotion data.

[0758] Subject: terminal

[0759] The device transmits video and audio from the video conference to the server. The device uses its camera and microphone to capture participants' facial expressions and speech. This data is sent to the server in real time and analyzed by an AI engine.

[0760] Subject: User

[0761] Users participate in video conferences and send their speech and facial expressions to the server via their devices. Users simply participate in the meeting naturally without performing any special operations. The content of the user's speech, facial expressions, and tone of voice are analyzed by an AI engine to provide optimal meeting facilitation.

[0762] Specific example

[0763] Example 1: Analysis of speech during a meeting

[0764] When User A says, "What is the progress of this project?", the server uses speech recognition technology to convert the statement into text. The AI ​​engine then analyzes this text and recognizes that User A is asking a question. The server then facilitates the discussion, prompting other attendees to provide appropriate answers.

[0765] Example 2: Facial expression analysis

[0766] The terminal's camera captures user B's facial expression and sends it to the server. The server uses image recognition technology to analyze user B's facial expression and detects that user B is confused. The server pauses the meeting and facilitates the discussion by asking user B questions or comments.

[0767] Example of a prompt

[0768] Please describe a system that analyzes participants' speech, facial expressions, and tone of voice during video conferences to provide optimal meeting facilitation. Please include specific hardware and software names.

[0769] In this way, the system's operation can be explained from the perspectives of the server, terminal, and user, and understanding can be deepened with concrete examples. This system makes it possible to analyze the emotions of attendees in real time and achieve optimal meeting management. Furthermore, by providing feedback to improve the quality and effectiveness of meetings, it reduces the burden on users and enables smooth communication even in remote settings.

[0770] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0771] Step 1: Start the video conference

[0772] Subject: User

[0773] The user starts a video conference using their device. The user launches the meeting application, enters the meeting ID, and joins the meeting. For example, if a user joins a meeting using a meeting app, they enter the meeting ID and click the "Join" button.

[0774] Input: Meeting ID, User actions

[0775] Output: Start video conference, prepare video and audio capture.

[0776] Step 2: Capture

[0777] Subject: terminal

[0778] The device uses a camera and microphone to capture the user's video and audio. The camera captures the user's facial expressions in real time, and the microphone records what the user says. For example, the device's camera captures the user's smile, and the microphone records them saying "hello."

[0779] Input: User video and audio

[0780] Output: Captured video and audio data

[0781] Step 3: Data transmission

[0782] Subject: terminal

[0783] The device sends the captured video and audio data to the server. The data is encrypted and sent securely to the server. For example, the device sends the captured video and audio data to the server using the HTTPS protocol.

[0784] Input: Captured video and audio data

[0785] Output: Data sent to the server

[0786] Step 4: Data Analysis

[0787] Subject: Server

[0788] The server analyzes the received video and audio data. It uses speech recognition technology to convert the audio data into text and image recognition technology to analyze facial expressions from the video data. For example, the server uses speech recognition technology to convert the audio "hello" into text and image recognition technology to detect the user's smile.

[0789] Input: Video and audio data sent to the server

[0790] Output: Text data, facial expression analysis results

[0791] Step 5: Quantifying emotions

[0792] Subject: Server

[0793] The server quantifies the user's emotions based on the analysis results. The AI ​​engine analyzes changes in voice tone and facial expressions to generate an emotion score. For example, the server might quantify the user's smile as "joy" and assign an emotion score of 80 / 100.

[0794] Input: Text data, facial expression analysis results

[0795] Output: Sentiment score

[0796] Step 6: Facilitating

[0797] Subject: Server

[0798] The server uses quantified sentiment data to provide optimal meeting facilitation. For example, if a user appears confused, the server pauses the meeting and prompts the user to ask questions or offer opinions.

[0799] Input: Sentiment score

[0800] Output: Facilitation instructions

[0801] Step 7: Provide feedback

[0802] Subject: Server

[0803] The server provides feedback based on the progress of the meeting and sentiment data. For example, after the meeting ends, it sends attendees a report about changes in their sentiment and the frequency of their comments during the meeting.

[0804] Input: Meeting progress, sentiment data

[0805] Output: Feedback Report

[0806] By specifically describing each processing step and clearly indicating the inputs and outputs, the system's operation can be understood more clearly.

[0807] (Application Example 1)

[0808] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0809] Conventional video conferencing systems have struggled to analyze participants' speech, facial expressions, and tone of voice to quantify their emotions and facilitate meetings effectively. Furthermore, while efficient meetings and discussions within factories require efficient progress, there was a lack of automated methods to manage meetings while considering participants' emotions. This resulted in decreased meeting quality and effectiveness, and increased burden on users.

[0810] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0811] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for installation in a robot for streamlining meetings and discussions within a factory, and means for optimizing meeting progress while considering the emotions of attendees. This makes it possible to conduct efficient meetings and discussions within a factory while considering the emotions of attendees.

[0812] A "video conference" is a meeting conducted via the internet, where multiple participants share video and audio in real time.

[0813] "Facilitation" refers to the methods and techniques used to ensure that meetings and discussions proceed smoothly.

[0814] "AI" is an abbreviation for artificial intelligence, which is a technology that allows computers to learn and reason by mimicking human intelligence.

[0815] "Statements" refer to the information and opinions that participants express orally during meetings or discussions.

[0816] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[0817] "Voice quality" refers to the tone and texture of a voice, and is an element that reflects the speaker's emotions and intentions.

[0818] "Quantifying emotions" means analyzing the emotional state of participants and expressing it as numerical data.

[0819] "Optimal meeting facilitation" means taking into account the participants' comments and emotional states to conduct the meeting in the most effective way possible.

[0820] A "factory meeting" refers to a meeting held to discuss matters related to the operation and production of the factory.

[0821] A "meeting" is when several people gather to exchange opinions and share information regarding a specific purpose or issue.

[0822] A "robot" is a mechanical device that operates automatically according to a program and performs specific tasks.

[0823] "Installation" refers to bringing software or applications into a computer or machine and making them usable.

[0824] "Optimizing meeting progress" means making the necessary adjustments and improvements to ensure that meetings are conducted efficiently and effectively.

[0825] As an embodiment of this invention, we describe a "smart meeting robot" system for streamlining meetings and discussions within a factory. This system uses AI to automatically facilitate video conferences, analyzing the content of participants' statements, facial expressions, and tone of voice to quantify their emotions and perform optimal meeting facilitation.

[0826] System Configuration

[0827] 1. Hardware:

[0828] Camera: Used to capture attendees' facial expressions in real time.

[0829] Microphone: Used to collect the content of what attendees say.

[0830] Robot: A mechanical device that moves around within a factory and supports the progress of meetings.

[0831] 2. Software:

[0832] Speech recognition library: The speech_recognition library is used to recognize attendees' speech in real time.

[0833] Emotion analysis model: Using the transformers library's pipeline, the model analyzes the facial expressions and tone of voice of attendees and quantifies their emotions.

[0834] Image processing library: The cv2 library is used to acquire video from the camera in real time and perform facial expression analysis.

[0835] System operation

[0836] 1. Speech recognition:

[0837] The server converts the audio data acquired from the microphone into text data using the speech_recognition library.

[0838] The converted text data will be used as information to optimize the meeting's progress.

[0839] 2. Emotion analysis:

[0840] The server processes the video data acquired from the camera in real time using the cv2 library and analyzes the facial expressions of the attendees.

[0841] The analyzed facial expression data is then converted into numerical emotion using the transformers library's pipeline.

[0842] 3. Optimizing meeting facilitation:

[0843] The server executes logic to optimize the meeting's progress based on the acquired audio and emotion data.

[0844] For example, if an attendee says, "There's a problem with this part," the robot will respond, "A problem has occurred. I will suggest a solution."

[0845] Additionally, if an attendee's facial expression is negative, the message "Attendees are feeling negative. We will adjust the meeting's progress." will be displayed.

[0846] Specific example

[0847] As a concrete example, let's consider a quality control meeting in a factory. If an attendee says, "There is a problem with this part," the robot will respond, "A problem has occurred. I will propose a solution." Also, if the attendee's expression is negative, it will display, "The attendee's mood is negative. I will adjust the meeting's progress."

[0848] Example of a prompt

[0849] Examples of prompt statements to input into a generative AI model include the following:

[0850] If someone says, "There's a problem with this part," please suggest how the meeting should proceed.

[0851] In this way, the smart meeting robot system can streamline meetings within factories and provide optimal facilitation that takes into account the emotions of the attendees.

[0852] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0853] Step 1:

[0854] The server acquires video data from the camera in real time. The input is video data from the camera, and the output is frame data processed using the image processing library cv2. Specifically, the server captures the video from the camera frame by frame, converts it to grayscale, and performs face detection.

[0855] Step 2:

[0856] The server acquires audio data from the microphone in real time. The input is audio data from the microphone, and the output is the spoken content converted into text data using the speech recognition library speech_recognition. Specifically, the server records the audio from the microphone and converts it to text using the speech recognition engine.

[0857] Step 3:

[0858] The server analyzes the facial expressions of attendees using the acquired frame data. The input is the frame data obtained in step 1, and the output is emotion data quantified using the pipeline of the emotion analysis model transformers. Specifically, the server extracts the region where a face was detected and inputs it into the emotion analysis model to quantify the emotion.

[0859] Step 4:

[0860] The server analyzes the acquired text data and extracts information necessary for the meeting's progress. The input is the text data obtained in step 2, and the output is instructions and suggestions for the meeting's progress. Specifically, the server inputs the text data into a natural language processing engine and extracts important keywords and phrases.

[0861] Step 5:

[0862] The server executes logic to optimize the meeting's progress based on sentiment data and text data. The inputs are the sentiment data obtained in step 3 and the text data obtained in step 4, and the output is specific actions for managing the meeting. Specifically, the server integrates the sentiment data and text data and generates instructions to adjust the meeting's progress.

[0863] Step 6:

[0864] The server sends the generated instructions to the robot to support the progress of the meeting. The input is the meeting progress instructions obtained in step 5, and the output is the specific actions taken by the robot. Specifically, the server sends voice and motion instructions to the robot to support the progress of the meeting.

[0865] Step 7:

[0866] The user conducts the meeting by following instructions from the robot. The input is the instructions from the robot, and the output is the progress of the meeting. Specifically, the user speaks and acts according to the robot's instructions to ensure the meeting runs smoothly.

[0867] (Example 2)

[0868] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0869] Traditional video conferencing systems often require manual facilitation, placing a heavy burden on attendees and potentially reducing the quality and effectiveness of meetings. Furthermore, discussions related to patents require specialized knowledge, making it difficult to generate appropriate questions and suggestions. Additionally, non-face-to-face communication is often inefficient, leading to delays in meeting progress.

[0870] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0871] In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice and quantifying their emotions, means for optimal conference facilitation, means for collecting patent-related data, means for pre-processing the collected data, means for generating questions and suggestions using a generative AI model, means for transmitting the generated questions and suggestions to a terminal, and means for the terminal to present the results to the user. This reduces the burden on the user and improves the quality and effectiveness of meetings. It also facilitates specialized discussions related to patents and enables smooth communication even in non-face-to-face settings.

[0872] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[0873] "Facilitation" refers to activities that provide support and coordination to ensure the smooth progress of meetings and discussions.

[0874] "Artificial intelligence" is a technology in which computer systems imitate human intelligence to learn, reason, and self-correct.

[0875] "Statements" refer to the information and opinions that participants express orally during a meeting or discussion.

[0876] "Facial expression" refers to emotions and reactions expressed through the movement of facial muscles.

[0877] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and pitch.

[0878] "Quantifying emotions" means expressing emotional states as numerical data.

[0879] A "patent" is a legal right that grants an inventor the exclusive right to use their invention for a certain period of time.

[0880] "Collecting data" is the act of gathering necessary information.

[0881] "Preprocessing" refers to the initial processing required to prepare data into a format that is easy to analyze and use.

[0882] A "generative AI model" is a model that uses artificial intelligence to generate new data and information.

[0883] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific theme.

[0884] A "terminal" is a device used by a user to input or output information.

[0885] A "user" is a person who uses a system or service.

[0886] "Presenting results" means showing the generated information or data to the user.

[0887] This invention is a system that uses artificial intelligence to automatically facilitate video conferences. The system has the function of analyzing the content of participants' speech, facial expressions, and tone of voice, and quantifying their emotions. It also includes a function to collect patent-related data, perform preprocessing, and generate questions and suggestions using a generative AI model. A specific embodiment of this system is described below.

[0888] Hardware and software configuration

[0889] server

[0890] The server is equipped with the hardware and software necessary to perform the following functions:

[0891] Data collection: The server collects the necessary information from patent-related databases (e.g., patent information databases).

[0892] Data preprocessing: The server uses Python's NLTK library and spaCy to preprocess the collected data.

[0893] Application of Generative AI Models: The server uses a pre-trained generative AI model (e.g., GPT-4) to generate patent-related questions and suggestions.

[0894] Sending results: The server sends the generated questions and suggestions to the terminal.

[0895] terminal

[0896] A terminal is a device used by users to input information and display results sent from a server. A terminal has the following functions:

[0897] Receiving user input: The terminal receives prompts entered by the user.

[0898] Displaying results: The terminal presents the user with questions and suggestions sent from the server.

[0899] User

[0900] A user is someone who uses the system to conduct a video conference. The user performs the following actions:

[0901] Prompt input: The user enters prompts into the terminal to facilitate discussions related to the patent.

[0902] Specific example

[0903] For example, consider a scenario where a user is conducting a meeting about patents.

[0904] Example of a prompt:

[0905] "We would like to discuss a new patent application. Please generate questions and suggestions regarding the patent's technical details and the competitive landscape of the market."

[0906] When the user enters this prompt into the terminal, the server collects relevant data from the patent information database and performs preprocessing. Next, it uses a generative AI model to generate appropriate questions and suggestions and sends them to the terminal. The terminal then presents the generated questions and suggestions to the user. For example, a question such as "How does this technology differ from existing patents?" might be displayed.

[0907] In this way, users can use the system to efficiently advance discussions related to patents. The system can reduce the burden on users and improve the quality and effectiveness of meetings. It can also enable smooth communication even when meetings are not face-to-face.

[0908] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0909] Step 1:

[0910] The user enters a prompt message.

[0911] The user enters prompts into the terminal to facilitate discussions related to the patent. For example, they might enter, "I would like to discuss a new patent application. Please generate questions and suggestions regarding the patent's technical details and market competitive landscape." The entered prompts are then sent from the terminal to the server.

[0912] Step 2:

[0913] The server collects the data.

[0914] The server receives prompt messages from the user and collects patent-related data. Specifically, it retrieves relevant patent information from a patent information database. For example, it calls the API of the patent information database to collect patent information related to "new technologies." The input is the prompt message, and the output is the collected patent information.

[0915] Step 3:

[0916] The server preprocesses the data.

[0917] The server preprocesses the collected patent information. Using Python's NLTK library and spaCy, it analyzes the text data and extracts important keywords and phrases. For example, it extracts keywords such as "technical details" and "market competitive landscape." The input is the collected patent information, and the output is the preprocessed data.

[0918] Step 4:

[0919] The server applies the generated AI model.

[0920] The server generates questions and suggestions using a generative AI model (e.g., GPT-4) based on pre-processed data. The generative AI model generates appropriate questions and suggestions based on extracted keywords. For example, it might generate a question such as, "How does this technology differ from existing patents?" The input is pre-processed data, and the output is the generated questions and suggestions.

[0921] Step 5:

[0922] The server sends the generated results to the terminal.

[0923] The server sends the generated questions and suggestions to the terminal. The generated questions and suggestions are sent to the terminal in JSON format. For example, they are sent in the format {"question": "How does this technology differ from existing patents?"}. The input is the generated questions and suggestions, and the output is the data sent to the terminal.

[0924] Step 6:

[0925] The device presents the results to the user.

[0926] The terminal presents the user with questions and suggestions received from the server. The user can then proceed with the discussion based on the presented questions and suggestions. For example, the terminal screen might display "How does this technology differ from existing patents?". The input is data sent from the server, and the output is the questions and suggestions presented to the user.

[0927] (Application Example 2)

[0928] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0929] Traditional video conferencing systems require manual facilitation, which places a heavy burden on attendees and can lead to a decline in the quality and effectiveness of meetings. Furthermore, discussions related to patents and technical debates require specialized knowledge, making it difficult to ask appropriate questions or make suggestions. Additionally, the inability to facilitate smooth non-face-to-face communication was a problem.

[0930] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0931] In this invention, the server includes means for an AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for generating questions and suggestions to promote discussions related to patents, and means for generating prompt sentences based on the meeting topic using a generative AI model. This improves the quality and effectiveness of meetings, efficiently supports discussions and technical debates related to patents, and enables smooth communication even when not face-to-face.

[0932] A "video conference" is a meeting conducted via the internet, in which multiple participants share video and audio in real time.

[0933] "Facilitation" refers to activities that provide support and coordination to ensure that meetings and discussions proceed smoothly.

[0934] "AI" is an abbreviation for artificial intelligence, a technology in which computers imitate human intelligence to perform learning and reasoning.

[0935] "Statements" refer to the information and opinions expressed orally by participants during meetings or discussions.

[0936] "Facial expression" refers to emotions and reactions expressed through the movement of facial muscles.

[0937] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and intonation.

[0938] "Quantifying emotions" means expressing emotional states as numerical data.

[0939] A "patent" is an exclusive right granted to an invention.

[0940] "To facilitate discussion" means to support participants in actively exchanging opinions and engaging in lively debate.

[0941] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific topic.

[0942] A "generative AI model" is an artificial intelligence model that generates new data or information based on input data.

[0943] A "prompt statement" refers to an instruction or question that is input into a generative AI model.

[0944] "Factory interior" refers to the inside of a facility where production activities take place in the manufacturing industry.

[0945] A "technical discussion" is a meeting where experts exchange opinions and debate technical matters.

[0946] A system for carrying out this invention includes a program for automatically facilitating video conferences. The system is implemented using the following hardware and software.

[0947] hardware

[0948] Server: A server with high-performance computing capabilities is required. This will enable rapid execution of generative AI models and data analysis.

[0949] Device: You will need a device (such as a PC, tablet, or smartphone) to participate in the video conference.

[0950] Cameras and microphones: Cameras and microphones are necessary to capture what attendees say, their facial expressions, and their tone of voice.

[0951] software

[0952] Generative AI Model: Uses the OpenAI API to generate prompt sentences based on the meeting topic.

[0953] Data analysis software: Software is needed to analyze the content of attendees' statements, facial expressions, and tone of voice, and to quantify their emotions.

[0954] Video conferencing software: You will need software to conduct video conferences (for example, Zoom or Microsoft Teams).

[0955] Data processing and data calculation

[0956] The server analyzes the participants' statements, facial expressions, and tone of voice captured during video conferences in real time, quantifying their emotions. This allows for understanding the progress of the meeting and the participants' reactions. Furthermore, using a generative AI model, it generates prompt sentences based on the meeting topic and automatically asks questions and makes suggestions to facilitate discussions related to patents.

[0957] Specific example

[0958] For example, when conducting a meeting about a patent for a new manufacturing process, the server generates a prompt message like the following:

[0959] "What technical features should we emphasize to obtain a patent for this manufacturing process?"

[0960] "What are the advantages of this process compared to the patents of our competitors?"

[0961] This allows attendees to conduct discussions more efficiently, improving the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[0962] In this way, the system for implementing the invention can automate the facilitation of video conferences and support patent-related discussions and technical debates.

[0963] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0964] Step 1:

[0965] The server activates the camera and microphone at the start of a video conference to capture participants' speech, facial expressions, and tone of voice. It receives real-time data from the camera and microphone as input and sends this data to analysis software as output.

[0966] Step 2:

[0967] The server analyzes the acquired speech content, facial expressions, and tone of voice data using analysis software to quantify emotions. It receives real-time data from the camera and microphone as input and generates quantified emotion data as output. Specifically, it uses speech recognition technology to transcribe speech into text and facial recognition technology to analyze facial muscle movements.

[0968] Step 3:

[0969] The server generates prompt sentences based on the meeting topic using a generative AI model, based on the analysis results. It receives quantified sentiment data and the meeting topic as input, and generates prompt sentences as output. Specifically, it uses the OpenAI API to generate the prompt sentences.

[0970] Step 4:

[0971] The server sends the generated prompt message to the video conferencing software and presents it to the attendees. It receives the generated prompt message as input and displays the prompt message in the video conferencing software as output. Specifically, it displays the prompt message using the chat or screen sharing functions of the video conferencing software.

[0972] Step 5:

[0973] The user advances the discussion based on the presented prompt. It receives the prompt as input and generates the discussion content as output. Specifically, the user provides opinions and questions in response to the prompt.

[0974] Step 6:

[0975] The server monitors the progress of the discussion and generates additional prompts as needed. It receives discussion content and sentiment data as input and generates additional prompts as output. Specifically, it uses the generative AI model again to generate new prompts.

[0976] Step 7:

[0977] The server saves all data and generates a meeting report at the end of the meeting. It receives all data acquired during the meeting as input and generates a meeting report as output. Specifically, it saves the data to a database and creates the report using report generation software.

[0978] (Example 3)

[0979] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0980] Traditional video conferencing systems struggled to analyze participants' emotions in real time and appropriately adjust the meeting's progress. This resulted in decreased meeting quality and effectiveness, and increased user burden. Furthermore, the inability to communicate effectively without face-to-face interaction often led to delays in meetings.

[0981] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for artificial intelligence to automatically facilitate the meeting, means for analyzing the content of attendees' statements, facial expressions, and tone of voice and quantifying their emotions, means for optimal meeting facilitation, means for collecting meeting audio data, means for transmitting the collected audio data to the server, means for the server to perform emotion analysis on the audio data, means for adjusting the progress of the meeting based on the analysis results, and means for suggesting appropriate actions to the user. This improves the quality and effectiveness of meetings, reduces the burden on users, and enables smooth communication even when not face-to-face.

[0982] "Methods for using artificial intelligence to automatically facilitate meetings" refers to functions that use artificial intelligence to automatically manage and coordinate meetings.

[0983] "A means of analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to a function that analyzes the statements, facial expressions, and tone of voice of attendees and expresses those emotions as numerical values.

[0984] "A means of optimally facilitating meetings" refers to a function that makes adjustments to optimize the progress of meetings based on analyzed emotional data.

[0985] "Means for collecting meeting audio data" refers to a function for collecting audio during a meeting in real time.

[0986] "Means for sending collected audio data to a server" refers to a function for sending collected audio data to a server via the internet.

[0987] "Means for a server to analyze the emotions of audio data" refers to a function in which the server analyzes the received audio data to identify the emotions of the attendees.

[0988] "Means for adjusting the progress of a meeting based on analysis results" refers to a function that appropriately adjusts the progress of a meeting based on the results of sentiment analysis.

[0989] "Means of suggesting appropriate actions to users" refers to functions that suggest appropriate actions to users based on analysis results.

[0990] This invention relates to a system that uses artificial intelligence to automatically facilitate meetings. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to enable optimal meeting facilitation.

[0991] 1. Generate the system program.

[0992] This system's program is developed using Python. The main libraries used are "Hugging Face Transformers" for sentiment analysis and the "Google Calendar API" for meeting management.

[0993] 2. Explain the program's processing in natural language.

[0994] When a user starts a meeting, the device collects the meeting's audio data in real time. The collected audio data is sent to a server, where sentiment analysis is performed using Hugging Face Transformers. The analysis results are categorized into three categories: positive, negative, and neutral.

[0995] The server adjusts the meeting's progress based on the analysis results. For example, if it determines that an attendee's emotions are leaning towards negative, the server will pause the meeting via the Google Calendar API and display a message on the device suggesting a break.

[0996] 3. Add specific examples to the explanation.

[0997] The following scenario is a concrete example.

[0998] scenario:

[0999] A user starts an online meeting. During the meeting, one of the attendees repeatedly makes negative remarks. The system performs sentiment analysis on the remarks and determines that the negative emotions are strong. The server pauses the meeting and displays the message "Do you want to suggest taking a break?" on the user's device.

[1000] Example of a prompt:

[1001] "Develop a system that analyzes participants' emotions in real time during online meetings and suggests pausing the meeting and taking a break if negative emotions are strongly present."

[1002] This system provides an environment that enables smooth communication even without face-to-face interaction. The flow of a specific process in Example 3 will be explained using Figure 15.

[1003] Step 1:

[1004] The user initiates the meeting. The user launches the meeting application and clicks the "Start Meeting" button. This causes the system to begin collecting audio data. The input is the user's actions, and the output is the meeting start signal.

[1005] Step 2:

[1006] The device collects audio data from the meeting. The device uses a microphone to pick up audio during the meeting and converts it into digital audio data in real time. For example, if a user says "Hello everyone," that audio is immediately collected as digital data. The input is the audio from the meeting, and the output is digital audio data.

[1007] Step 3:

[1008] The terminal sends the collected audio data to the server. The terminal sends the collected audio data to the server using the HTTPS protocol. The data is AES encrypted and transmitted securely. The input is digital audio data, and the output is a signal indicating that transmission to the server is complete.

[1009] Step 4:

[1010] The server performs sentiment analysis on the audio data. The server inputs the received audio data into the Hugging Face Transformers sentiment analysis model. For example, the statement "This project might not work out" is analyzed as negative. The input is digital audio data, and the output is the sentiment analysis result.

[1011] Step 5:

[1012] The server adjusts the meeting's progress based on the analysis results. If the server determines that the sentiment analysis results are negative, it issues a command to temporarily suspend the meeting using the Google Calendar API. The input is the sentiment analysis result, and the output is a command to adjust the meeting's progress.

[1013] Step 6:

[1014] The terminal suggests appropriate actions to the user. The terminal receives instructions from the server and displays a message to the user saying, "Attendees are feeling negative. Would you like to suggest a break?" The user adjusts the meeting's progress by selecting "Yes" or "No." The input is the instruction from the server, and the output is the suggestion message to the user.

[1015] In this way, the system provides an environment where smooth communication can take place even without face-to-face interaction.

[1016] (Application Example 3)

[1017] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1018] Traditional video conferencing systems and work environments have struggled to adequately analyze the emotions of attendees and workers, thereby optimizing meeting progress and work efficiency. Furthermore, the inability to respond appropriately when emotions turned negative led to problems with reduced meeting quality and work safety. This resulted in increased user burden and hindered smooth non-face-to-face communication.

[1019] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means.

[1020] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for temporarily suspending work and suggesting a break if emotions become negative. This optimizes the progress of meetings and work efficiency, reduces the burden on users, and enables smooth communication even in non-face-to-face settings.

[1021] A "video conference" is a meeting conducted over the internet, where multiple participants share video and audio.

[1022] "Facilitation" refers to providing support and coordination to ensure that meetings and work proceed smoothly.

[1023] "AI" stands for artificial intelligence, which is a technology in which machines imitate human intelligence to learn and reason.

[1024] "Statements" refer to the information and opinions that participants express orally during meetings or work sessions.

[1025] "Facial expression" refers to the emotions and intentions conveyed through the movement of the facial muscles.

[1026] "Voice quality" refers to the sound quality and tone of a voice, and it reflects the speaker's emotions and state of mind.

[1027] "Quantifying emotions" means expressing emotions as quantitative data.

[1028] A "worker" is a person who performs a specific task in a factory, office, or similar setting.

[1029] "Real-time" means processing events that are currently unfolding immediately.

[1030] "Work efficiency" refers to the ability to minimize wasted time and effort during work and achieve maximum results.

[1031] "Safety" refers to a state in which work or activities can be carried out without danger and with peace of mind.

[1032] "Suggesting a break" means temporarily interrupting work or a meeting and encouraging participants to take a rest.

[1033] A "system" is a mechanism in which multiple elements work together to perform a specific function.

[1034] The system for implementing this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for temporarily suspending work and suggesting a break if emotions become negative.

[1035] System program

[1036] The program in this system performs the following operations:

[1037] 1. Hardware: The system uses a camera (e.g., Logitech C920) to capture the faces of attendees and workers.

[1038] 2. Software: The system uses software such as OpenCV, Keras, and TensorFlow to process captured images and analyze emotions.

[1039] 3. Data Processing: The captured images are converted to grayscale, and face detection is performed. The detected face regions are preprocessed for input into the emotion analysis model.

[1040] 4. Data processing: Use an emotion analysis model to predict emotions from pre-processed facial images.

[1041] Specific example of processing

[1042] For example, if a worker in a factory is experiencing stress, the system analyzes their emotions in real time, temporarily suspends their work, and suggests a break. This improves work efficiency and safety, and reduces the burden on workers.

[1043] Example of a prompt

[1044] Examples of prompts to input into a generative AI model are as follows:

[1045] Develop a robot assistant application that analyzes workers' emotions in real time within a factory to improve work efficiency and safety. Include a feature that allows the robot to pause work and suggest a break if a worker is experiencing stress.

[1046] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1047] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[1048] Step 1:

[1049] The server uses a camera to capture the faces of attendees and workers. The input is video data from the camera, and the output is the captured image data. Specifically, the camera acquires video in real time, and the server captures that video frame by frame.

[1050] Step 2:

[1051] The server converts the captured image data to grayscale. The input is the captured image data, and the output is grayscale image data. Specifically, it uses OpenCV to convert the image to grayscale.

[1052] Step 3:

[1053] The server detects faces from grayscale images. The input is grayscale image data, and the output is the coordinate data of the detected face region. Specifically, it uses the OpenCV face detection algorithm to identify the face region.

[1054] Step 4:

[1055] The server preprocesses the detected face regions for input into the emotion analysis model. The input consists of face region coordinate data and grayscale image data, and the output is the preprocessed face image data. Specifically, the server extracts the face region, resizes it to 48x48 pixels, and normalizes it.

[1056] Step 5:

[1057] The server inputs pre-processed facial image data into an emotion analysis model to predict emotions. The input is pre-processed facial image data, and the output is the predicted emotion. Specifically, Keras is used to input data into the emotion analysis model and obtain the prediction results.

[1058] Step 6:

[1059] The server determines whether the predicted emotion is negative. The input is the predicted emotion, and the output is the determination of whether it is a negative emotion. Specifically, it analyzes the prediction result and checks if it contains negative emotions (e.g., anger, sadness, fear).

[1060] Step 7:

[1061] The server pauses work and suggests a break if negative emotions are detected. The input is the result of the negative emotion detection, and the output is a notification suggesting a break. Specifically, the system pauses work and sends a notification to the user instructing them to take a break.

[1062] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1063] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1064] "Example of form 1"

[1065] In one embodiment of the present invention, an emotion engine recognizes the user's emotions and quantifies them. Specifically, it estimates emotions from the user's statements, tone of voice, facial expressions, etc., and quantifies them. This quantified emotion data is used to facilitate meetings. For example, if the AI ​​determines that the emotions of all participants are leaning towards negative, it takes actions such as pausing the meeting and suggesting a break.

[1066] "Example of form 2"

[1067] Furthermore, the emotion engine detects changes in users' emotions in real time and adjusts the meeting facilitation accordingly. For example, if the AI ​​senses that a participant is showing anger or dissatisfaction, it will take action such as providing that participant with an opportunity to speak or conveying that emotion to other participants. This ensures that the meeting proceeds smoothly and improves the overall satisfaction of the participants.

[1068] "Example of form 3"

[1069] Furthermore, the emotion engine predicts changes in users' emotions and optimizes meeting facilitation based on those predictions. For example, if it is predicted that a participant is likely to be dissatisfied with the meeting's progress, the AI ​​takes action such as providing that participant with an opportunity to speak in advance or adjusting the meeting's flow. This makes the meeting run more smoothly and further improves overall participant satisfaction.

[1070] The following describes the processing flow for each example of the form.

[1071] "Example of form 1"

[1072] Step 1: Estimate the user's emotions from their speech, tone of voice, facial expressions, etc.

[1073] Step 2: Quantify the estimated emotions.

[1074] Step 3: Utilize quantified emotional data to facilitate meetings.

[1075] Step 4: If the AI ​​determines that the overall sentiment of the participants is leaning towards negative, it will take action such as pausing the meeting and suggesting a break.

[1076] "Example of form 2"

[1077] Step 1: The emotion engine detects changes in the user's emotions in real time.

[1078] Step 2: Adjust the meeting facilitation according to changes in emotions.

[1079] Step 3: If the AI ​​senses that a participant is showing anger or dissatisfaction, it will take action such as giving that participant an opportunity to speak or conveying their feelings to other participants.

[1080] "Example of form 3"

[1081] Step 1: The emotion engine predicts changes in the user's emotions.

[1082] Step 2: Optimize meeting facilitation based on predictions.

[1083] Step 3: If the AI ​​predicts that a participant is likely to be dissatisfied with the meeting's progress, it will take action such as providing that participant with an opportunity to speak beforehand or adjusting the meeting's flow.

[1084] (Example 1)

[1085] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1086] Traditional video conferencing systems struggled to analyze participants' emotions and comments in real time and optimize meeting progress. Furthermore, there was a lack of means to provide professional facilitation to improve the quality and effectiveness of meetings. Additionally, there was no environment in place to reduce the burden on users and facilitate smooth communication in non-face-to-face settings.

[1087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1088] In this invention, the server includes means for capturing the user's speech and facial expressions in real time using a high-resolution camera and microphone; means for transmitting the captured audio and video data to the server; means for converting the audio data into text using speech recognition software; means for analyzing the user's facial expressions from the video data using image recognition software; means for quantifying the user's emotions from the text and facial data using an emotion analysis engine; and means for optimizing the progress of the meeting based on the quantified emotion data. This makes it possible to analyze the emotions and speech of attendees in real time and provide optimal meeting facilitation.

[1089] A "high-resolution camera" is a camera that can capture high-resolution video to capture the user's face and facial expressions in detail.

[1090] A "microphone" is an audio input device used to accurately record what a user says.

[1091] "Audio data" refers to audio information, including the content of the user's speech, captured through a microphone.

[1092] "Video data" refers to video information, including the user's face and facial expressions, acquired through a high-resolution camera.

[1093] A "server" is a computer system used to receive and analyze audio and video data.

[1094] "Speech recognition software" is software that analyzes speech data and converts it into corresponding text data.

[1095] "Text data" refers to character information generated from speech data by speech recognition software.

[1096] "Image recognition software" is software that analyzes video data to recognize the user's facial expressions.

[1097] An "emotion analysis engine" is an analysis system that estimates and quantifies a user's emotions based on text data and facial expression data.

[1098] "Emotional data" refers to user emotional information quantified by an emotion analysis engine.

[1099] "Methods for optimizing meeting progress" refer to methods for adjusting the progress of a meeting and providing optimal facilitation based on quantified emotional data.

[1100] This invention relates to a system that uses AI to automatically facilitate video conferences. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to provide optimal conference facilitation.

[1101] Hardware and software to be used

[1102] Hardware: High-resolution camera, microphone, server

[1103] Software: Speech recognition software (e.g., Google Speech-to-Text), image recognition software (e.g., OpenCV), sentiment analysis engine (e.g., IBM Watson)

[1104] System Overview

[1105] 1. Acquisition of user speech and facial expressions

[1106] The device uses a high-resolution camera and microphone to capture the user's speech and facial expressions in real time.

[1107] Specifically, the camera captures the user's face, and the microphone records the user's voice.

[1108] For example, if a user frowns and says, "This project is difficult," the camera and microphone capture that moment.

[1109] 2. Sending data

[1110] The device sends the captured audio and video data to the server.

[1111] Specifically, the device divides the data into packets and sends them to the server via the internet.

[1112] For example, user speech and facial expression data are sent to the server in real time.

[1113] 3. Speech Recognition and Image Recognition

[1114] The server uses speech recognition software to convert the audio data into text.

[1115] Simultaneously, the server uses image recognition software to analyze the user's facial expressions from the video data.

[1116] Specifically, speech recognition software analyzes the audio waveform and generates corresponding text. Image recognition software detects facial feature points and analyzes facial expressions.

[1117] For example, the statement "This project is difficult" is converted into text, and a frowning expression is analyzed as "confusion."

[1118] 4. Quantifying emotions

[1119] The server uses an emotion analysis engine to estimate the user's emotions from text data and facial expression data, and then quantifies them.

[1120] Specifically, the emotion analysis engine receives text and facial expression data as input and generates an emotion score.

[1121] For example, a confused expression and the statement "difficult" would result in a high negative emotion score.

[1122] 5. Meeting Facilitation

[1123] The server optimizes the meeting's progress based on quantified emotional data.

[1124] Specifically, the server analyzes the sentiment score, pauses the meeting as needed, and suggests a break.

[1125] For example, if the server determines that the overall sentiment score of the participants is leaning towards the negative, it will suggest, "Let's take a break."

[1126] Specific example

[1127] For example, the following scenario is possible.

[1128] Scenario: During the meeting, many participants looked tired, and their comments became increasingly negative.

[1129] process:

[1130] The device's camera and microphone capture the participants' facial expressions and speech.

[1131] The server uses speech recognition software to convert spoken content into text and image recognition software to analyze facial expressions.

[1132] The emotion analysis engine quantifies emotions from text and facial expression data and determines that there are many negative emotions.

[1133] The server pauses the meeting and suggests a break.

[1134] Example of a prompt

[1135] Examples of prompt statements to input into the generative AI model are as follows:

[1136] "If many participants in a meeting appear tired and their comments become increasingly negative, what are your suggestions for how to proceed with the meeting?"

[1137] By inputting this prompt into the AI ​​model, the AI ​​will suggest appropriate methods for facilitating meetings.

[1138] The flow of the specific processing in Example 1 will be explained using Figure 17.

[1139] Step 1:

[1140] The user speaks and shows facial expressions.

[1141] Input: User's statements and facial expressions.

[1142] Specific action: The user says, "This project is difficult," and frowns.

[1143] Step 2:

[1144] The device uses a high-resolution camera and microphone to capture the user's speech and facial expressions in real time.

[1145] Input: User's statements and facial expressions.

[1146] Output: Audio data and video data.

[1147] Specific operation: The camera captures the user's face, and the microphone records the user's voice.

[1148] Step 3:

[1149] The device sends the captured audio and video data to the server.

[1150] Input: Audio data and video data.

[1151] Output: Audio and video data sent to the server.

[1152] Specific operation: The terminal divides the data into packets and sends them to the server via the internet.

[1153] Step 4:

[1154] The server uses speech recognition software to convert the audio data into text.

[1155] Input: Audio data.

[1156] Output: Text data.

[1157] Specific operation: Speech recognition software analyzes the audio waveform and generates the corresponding text.

[1158] Step 5:

[1159] The server uses image recognition software to analyze the user's facial expressions from the video data.

[1160] Input: Video data.

[1161] Output: Facial expression data.

[1162] Specific operation: Image recognition software detects facial feature points and analyzes facial expressions.

[1163] Step 6:

[1164] The server uses an emotion analysis engine to quantify the user's emotions from text data and facial expression data.

[1165] Input: Text data and facial expression data.

[1166] Output: Sentiment data.

[1167] Specific operation: The emotion analysis engine receives text and facial expression data as input and generates an emotion score.

[1168] Step 7:

[1169] The server optimizes the meeting's progress based on quantified emotion data.

[1170] Input: Sentiment data.

[1171] Output: Instructions regarding the progress of the meeting.

[1172] Specific operation: The server analyzes the sentiment score, pauses the meeting as needed, and suggests a break.

[1173] (Application Example 1)

[1174] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1175] Conventional video conferencing systems and factory work support systems have struggled to adequately analyze the emotions of attendees and workers and provide optimal facilitation and work support based on that analysis. In particular, during long work sessions and meetings, stress and fatigue accumulate among participants and workers, leading to decreased efficiency and safety. Therefore, there is a need for new methods to improve the quality of meetings, work efficiency, and safety.

[1176] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1177] In this invention, the server includes means for an AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to support improvements in work efficiency and safety, means for suggesting breaks based on emotion recognition results, and means for notifying the suggestion by voice using a speech synthesis engine. This makes it possible to improve the efficiency and safety of meetings and work.

[1178] "Video conference facilitation" refers to the act of coordinating the progress of a video conference and encouraging participants to speak, thereby supporting the achievement of the meeting's objectives.

[1179] "Methods that AI automates" refers to methods or devices that use artificial intelligence to automatically perform specific tasks.

[1180] "Methods for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to methods or devices that use speech recognition technology or image recognition technology to analyze the content of attendees' statements, facial expressions, and tone of voice, and express the results as numerical values.

[1181] "Means for optimal meeting facilitation" refer to methods and devices for optimizing the progress of a meeting based on analyzed data.

[1182] "Means for analyzing workers' emotions in real time and providing support to improve work efficiency and safety" refers to methods and devices that analyze workers' emotions in real time and improve work efficiency and safety based on the results.

[1183] "Means for suggesting breaks based on emotion recognition results" refers to methods or devices for suggesting breaks to workers or attendees based on the results of emotion recognition.

[1184] "Means for notifying proposals by voice using a speech synthesis engine" refers to methods or devices for notifying proposal content by voice using speech synthesis technology.

[1185] The following system configuration is used as an embodiment of this invention. The server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to support improvements in work efficiency and safety, means for suggesting breaks based on emotion recognition results, and means for notifying the suggestion by voice using a speech synthesis engine.

[1186] Hardware and software to be used

[1187] Hardware: Webcam, microphone

[1188] Software: OpenCV, Keras, pyttsx3

[1189] Data processing and data calculation

[1190] The server acquires video and audio in real time using a webcam and microphone. The acquired video data is subjected to face recognition using OpenCV, and the facial region is extracted. The extracted facial region is input into a Keras emotion recognition model, and the emotion is quantified. The audio data is analyzed in the same way, and emotion is estimated from the tone of voice.

[1191] Based on the analyzed sentiment data, the server optimizes meeting facilitation. For example, if the overall sentiment of attendees is negative, it will pause the meeting and suggest a break. It will also suggest a break if a worker's sentiment indicates stress or fatigue. These suggestions are communicated via voice using a speech synthesis engine (pyttsx3).

[1192] Specific example

[1193] For example, consider a scenario where, in a factory, a worker is working for a long period of time, and an emotion recognition system detects the worker's stress level and suggests a break. In this case, the server uses a webcam and microphone to analyze the worker's facial expressions and tone of voice, and if it determines that the stress level is high, it notifies the worker via voice message saying, "Please take a break."

[1194] Examples of prompts to input into a generative AI model:

[1195] "Design a system that analyzes workers' emotions in real time within the factory and suggests breaks if stress or fatigue levels are high. Specifically, it would analyze workers' facial expressions and tone of voice, quantify their emotions, and make judgments based on that."

[1196] In this way, it becomes possible to improve the efficiency and safety of meetings and work.

[1197] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[1198] Step 1:

[1199] The server acquires video and audio in real time using a webcam and microphone. The input consists of video data from the webcam and audio data from the microphone. The output is the raw video and audio data acquired.

[1200] Step 2:

[1201] The server performs face recognition on the acquired video data using OpenCV. The input is the video data acquired in step 1. The output is image data with the face region extracted. Specifically, the server detects the position of the face from the video data and extracts that region.

[1202] Step 3:

[1203] The server inputs the extracted facial region into a Keras emotion recognition model to quantify the emotion. The input is the facial image data extracted in step 2. The output is numerical data indicating the emotion. Specifically, the facial image data is preprocessed and input into the emotion recognition model to predict the emotion.

[1204] Step 4:

[1205] The server analyzes the acquired audio data and estimates emotions from the tone of voice. The input is the audio data acquired in step 1. The output is numerical data indicating emotions based on the tone of voice. Specifically, the audio data is preprocessed, and audio features are extracted to estimate emotions.

[1206] Step 5:

[1207] The server integrates the emotion data obtained in steps 3 and 4 to perform an overall emotion assessment. The inputs are facial emotion data and vocal emotion data. The output is numerical data representing the overall emotion assessment. Specifically, it integrates the facial and vocal emotion data to calculate an overall emotion score.

[1208] Step 6:

[1209] The server provides optimal meeting facilitation and work support based on an overall sentiment assessment. The input is the overall sentiment assessment data obtained in step 5. The output is a specific action (e.g., suggesting a break). Specifically, if the sentiment assessment is leaning towards negative, the server will pause the meeting and suggest a break.

[1210] Step 7:

[1211] The server uses a speech synthesis engine (pyttsx3) to announce the proposal verbally. The input is the action determined in step 6. The output is the verbal notification. Specifically, it notifies attendees and workers verbally of the suggestion to take a break.

[1212] In this way, the server can analyze emotions in real time and provide optimal meeting facilitation and work support.

[1213] (Example 2)

[1214] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1215] Traditional video conferencing systems often require manual meeting management and facilitation, placing a significant burden on attendees and potentially reducing the quality and effectiveness of meetings. Furthermore, it was difficult to gauge attendees' emotions in real time and respond accordingly, hindering smooth communication. Additionally, generating appropriate questions and suggestions was challenging during specialized discussions, such as those focused on patents or strategies.

[1216] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice and quantifying their emotions, means for optimal meeting facilitation, means for generating relevant questions and suggestions based on the theme and purpose of the meeting, and means for adjusting the progress of the meeting based on emotion data. This makes it possible to reduce the burden on attendees and improve the quality and effectiveness of the meeting. In addition, real-time responses based on emotion data enable smooth communication. Furthermore, even in specialized discussions such as patent-focused or strategy-focused discussions, the depth and efficiency of the discussion can be improved by automatically generating appropriate questions and suggestions.

[1217] A "video conference" is a type of meeting in which multiple participants communicate in real time using audio and video over the internet.

[1218] "Facilitation" is the process of providing support and coordination to ensure that meetings and discussions proceed smoothly.

[1219] Artificial intelligence is a computer system that imitates human intelligence and performs learning, reasoning, and problem-solving.

[1220] "Statements" refer to the information and opinions that participants express orally during a meeting.

[1221] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[1222] "Voice quality" refers to the characteristics of a voice, such as tone, pitch, and rhythm, and is an element that conveys the speaker's emotions and intentions.

[1223] "Quantifying emotions" means expressing analyzed emotional data as quantitative numerical values.

[1224] "Optimal meeting facilitation" means conducting and coordinating meetings in the most effective way, according to their purpose and circumstances.

[1225] The "meeting theme" refers to the main topics or issues that will be discussed at the meeting.

[1226] "Purpose" refers to the specific goals or outcomes that the meeting aims to achieve.

[1227] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on the meeting's theme and objectives.

[1228] "Emotional data" refers to the analysis results that show the emotional state of the participants.

[1229] "Adjusting the progress of a meeting" means managing the flow and progress of a meeting appropriately and making adjustments to ensure it runs smoothly.

[1230] A "patent-focused" conference is a conference format specifically designed for discussions and examinations related to patents.

[1231] A "strategy-focused" meeting is a conference format specifically designed for discussions and considerations related to strategy.

[1232] Modes for carrying out the invention

[1233] This invention is a system that uses artificial intelligence to automatically facilitate video conferences, and it has the function of analyzing the content of participants' speech, facial expressions, and tone of voice to quantify their emotions. It also includes a function to generate relevant questions and suggestions based on the theme and purpose of the meeting, and to adjust the progress of the meeting based on the emotion data.

[1234] System Configuration

[1235] This system consists of three main elements: servers, terminals, and users.

[1236] server

[1237] The server uses a generative AI model to generate questions and suggestions based on the meeting's theme and objectives. For example, OpenAI's GPT-4 is used as the generative AI model. The server receives input from the user, generates appropriate questions and suggestions based on that input, and sends them to the terminal. The server also receives sentiment data sent from the terminal and determines actions to adjust the meeting's progress.

[1238] terminal

[1239] The device uses its camera and microphone to detect the user's emotions in real time. For example, it uses Microsoft's Azure Cognitive Services Emotion API as its emotion engine. The device captures the user's facial expressions through the camera and records their voice tone through the microphone. This data is sent to the emotion engine, where the user's emotions are analyzed in real time. The device receives instructions from the server and adjusts the meeting's progress accordingly.

[1240] User

[1241] Users use a terminal to input the meeting topic and purpose. For example, they might input, "I want to discuss new patent ideas." If a user expresses anger or dissatisfaction during the meeting, the terminal sends that information to the server, which then uses that information to determine what actions to take to adjust the meeting's progress.

[1242] Specific example

[1243] As a concrete example, let's consider the case of patent-focused facilitation. If a user inputs "I want to discuss a new patent idea," the server uses a generative AI model to automatically generate questions such as "What is the technical novelty of this idea?" and "How does it differ from existing patents?"

[1244] As a concrete example of an emotion engine, consider a situation where a participant expresses dissatisfaction during a meeting. The terminal analyzes the participant's facial expressions and tone of voice to detect dissatisfaction. The server receives this information and instructs the terminal to provide an opportunity for the participant to speak, such as by asking, "Do you have any comments, Mr. / Ms. XX?"

[1245] Example of a prompt

[1246] "I'd like to discuss new patent ideas."

[1247] "Please generate questions about the novelty of this technology."

[1248] "Please suggest appropriate responses to participants who express dissatisfaction during the meeting."

[1249] In this way, the system can reduce the burden on users and improve the quality and effectiveness of meetings.

[1250] The flow of the specific processing in Example 2 will be explained using Figure 19.

[1251] Step 1:

[1252] The user enters the meeting topic and purpose. The user uses the terminal's keyboard to type, "I want to discuss new patent ideas." This input data is sent from the terminal to the server.

[1253] Step 2:

[1254] The server uses a generative AI model to generate questions and suggestions related to the meeting topic. The server inputs the topic and objectives received from the user into the generative AI model and generates questions such as "What is the technical novelty of this idea?" or "How does it differ from existing patents?" The generated questions are sent from the server to the terminal.

[1255] Step 3:

[1256] The device detects the user's emotions in real time. The device captures the user's facial expressions through the camera and records their voice tone through the microphone. This data is sent to an emotion engine, which analyzes the user's emotions in real time. For example, if the user is frowning, anger is detected. This emotion data is then sent from the device to a server.

[1257] Step 4:

[1258] The server receives sentiment data and determines actions to adjust the meeting's progress. For example, if a user expresses dissatisfaction, the server instructs the terminal to give that user an opportunity to speak. This instruction is sent from the server to the terminal.

[1259] Step 5:

[1260] The terminal follows instructions from the server and coordinates the progress of the meeting. The terminal receives instructions from the server and displays a message to the user saying, "Do you have any comments, [Username]?" When the user begins to speak, the terminal sends the content to the server, ensuring the meeting proceeds smoothly.

[1261] (Application Example 2)

[1262] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1263] Traditional video conferencing systems have struggled to detect participants' emotions in real time and adjust the meeting's progress accordingly. Furthermore, there has been a lack of specialized facilitation services tailored to specific patent or strategy areas, making it difficult to improve the quality and effectiveness of meetings. Additionally, there is a need for methods to appropriately manage participants' emotions and facilitate smooth communication in factory meetings and brainstorming sessions.

[1264] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1265] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for supporting meetings and brainstorming sessions within a factory, and means for detecting participants' emotions in real time using an emotion engine and adjusting the progress of the meeting. This makes it possible to detect participants' emotions in real time and adjust the progress of the meeting based on that, enabling the provision of specialized facilitation focused on patents or strategies. Furthermore, it makes it possible to appropriately manage participants' emotions and promote smooth communication in meetings and brainstorming sessions within a factory.

[1266] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[1267] "Facilitation" is the activity of facilitating the smooth progress of meetings and group discussions, and supporting all participants in effectively exchanging opinions.

[1268] "AI" is an abbreviation for artificial intelligence, a technology in which computers mimic human intelligence to perform functions such as learning, reasoning, and recognition.

[1269] "Statements" refer to the opinions and information that participants express orally during a meeting or discussion.

[1270] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[1271] "Voice quality" refers to the characteristics of a speaker's voice, such as its tone, rhythm, and other features, and is an element that conveys emotions and intentions.

[1272] "Quantifying emotions" means analyzing the emotional state of participants and expressing it as numerical data.

[1273] A "factory meeting" refers to a meeting held on the factory floor in a manufacturing company, where discussions take place regarding product production, quality control, and operational improvements.

[1274] A "brainstorming session" is a discussion format in which participants freely share ideas and find creative solutions.

[1275] An "emotion engine" is a technology that analyzes data such as participants' facial expressions and tone of voice to detect their emotional state in real time.

[1276] "Real-time detection" means analyzing data instantly and obtaining results without delay.

[1277] "To coordinate the progress of a meeting" means to manage the flow and progress of the meeting and intervene or make corrections as needed.

[1278] "Patent-focused facilitation" refers to specialized support activities aimed at facilitating discussions related to patents.

[1279] "Strategy-focused facilitation" refers to specialized support activities aimed at facilitating discussions on strategic matters within companies and organizations.

[1280] The system for carrying out this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for supporting meetings and brainstorming sessions within factories, and means for detecting participants' emotions in real time using an emotion engine and adjusting the progress of the meeting.

[1281] The server first captures video of meeting participants in real time using a camera (e.g., Logitech C920). Next, it uses an emotion detection engine called EmotionEngine to analyze the participants' emotions from the captured video and express them as numerical data. The analyzed emotion data is sent to a facilitation support AI called FacilitationAI, which proposes actions to optimize the progress of the meeting.

[1282] Specifically, if one participant expresses anger or dissatisfaction, FacilitationAI instructs the server to give that participant an opportunity to speak. Conversely, if a participant expresses joy or satisfaction, it suggests actions to further encourage their participation. This leads to smoother meeting progress and increased overall participant satisfaction.

[1283] For example, if a meeting is being held in a factory to discuss the introduction of a new product line, and one of the participants expresses dissatisfaction, the server can provide that participant with an opportunity to speak and convey their feelings to the other participants, thereby smoothing the progress of the meeting.

[1284] Examples of prompts for a generative AI model include the following:

[1285] "We are holding a meeting about introducing a new product line. One of the participants is expressing dissatisfaction. Please suggest what facilitation actions should be taken in this situation."

[1286] In this way, the server can detect participants' emotions in real time and adjust the meeting's progress accordingly. This enables the provision of specialized facilitation focused on patents or strategies, and allows for appropriate management of participants' emotions and smooth communication in meetings and brainstorming sessions within factories.

[1287] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[1288] Step 1:

[1289] The server captures video of meeting participants in real time using a camera (e.g., Logitech C920). The input is video data from the camera, and the output is the captured video frame. This video frame is used in subsequent processing steps.

[1290] Step 2:

[1291] The server uses EmotionEngine to analyze participants' facial expressions from captured video frames and quantify their emotions. The input is video frames, and the output is numerical data indicating each participant's emotional state. Specifically, EmotionEngine analyzes the video frames, detects facial feature points, and classifies the emotions.

[1292] Step 3:

[1293] The server sends the analyzed sentiment data to FacilitationAI, which then proposes actions to optimize the meeting's progress. The input is sentiment data, and the output is the proposed facilitation action. Specifically, FacilitationAI analyzes the sentiment data and determines the appropriate action (e.g., provide opportunities to speak, encourage participation).

[1294] Step 4:

[1295] The server executes the actions suggested by FacilitationAI. The input is the suggested action, and the output is the result of the executed action. Specifically, the server manages the flow of the meeting, provides participants with opportunities to speak as needed, and communicates emotional states to other participants.

[1296] Step 5:

[1297] The server monitors the progress of the meeting and, if necessary, uses EmotionEngine again to detect participants' emotions and send them to FacilitationAI. The input is the progress of the meeting and the latest video frame, and the output is updated emotion data and new facilitation actions. Specifically, the server periodically captures video frames and re-analyzes the emotions to optimize the meeting's progress.

[1298] Step 6:

[1299] The server saves all sentiment data and a history of facilitation actions after a meeting to help improve future meetings. The input is sentiment data and action history, and the output is the saved data. Specifically, the server saves the sentiment data and action history to a database, which is then used as reference material for the next meeting.

[1300] (Example 3)

[1301] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1302] Traditional video conferencing systems have made it difficult to grasp the emotional state of attendees in real time and appropriately adjust the meeting's progress. Furthermore, they were unable to predict changes in attendees' emotions and optimize the meeting's progress based on those predictions, which sometimes led to a decline in the quality and effectiveness of meetings. Additionally, non-face-to-face communication was often not smooth, increasing the burden on users.

[1303] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1304] In this invention, the server includes means for collecting attendee data, means for analyzing the collected data and evaluating the emotional state of the attendees, means for adjusting the progress of the meeting based on the evaluation results, means for predicting changes in the emotional state of the attendees and optimizing the progress of the meeting based on the predictions, and means for providing smooth communication even when not face-to-face. This makes it possible to grasp the emotional state of attendees in real time and automatically perform appropriate facilitation. Furthermore, by predicting changes in the emotional state of attendees and optimizing the progress of the meeting, the quality and effectiveness of the meeting can be improved. In addition, by providing smooth communication even when not face-to-face, the burden on users can be reduced.

[1305] "Means for collecting attendee data" refers to devices or software that collect data such as facial expressions and voices of attendees participating in a meeting in real time.

[1306] "Means for analyzing collected data and evaluating the emotional state of attendees" refers to algorithms or software that analyze collected facial and audio data to evaluate the emotional state of attendees (joy, anger, sadness, etc.).

[1307] "Means for adjusting the progress of a meeting based on evaluation results" refers to systems or processes for temporarily pausing a meeting or suggesting a break based on the results of an evaluation of emotional state.

[1308] "Methods for predicting changes in attendees' emotions and optimizing the progress of a meeting based on those predictions" refers to algorithms or software that predict changes in attendees' emotions based on past emotional data and adjust the progress of the meeting based on those predictions.

[1309] "Means of providing smooth communication even without face-to-face interaction" refers to systems and processes that enable attendees to communicate smoothly with each other even when they are not physically meeting face-to-face.

[1310] This invention is a system for facilitating non-face-to-face communication and includes an AI function for automatically adjusting the progress of meetings. Specific embodiments of this system are described below.

[1311] As soon as the meeting starts, the server collects facial expressions and audio data from attendees via cameras and microphones connected to their devices (PCs or smartphones). The devices then transmit this data to the server in real time. The hardware used includes high-resolution cameras, microphones, and the server. The software used includes emotion analysis engines (e.g., Microsoft Azure's Emotion API) and meeting management systems (e.g., Zoom or Microsoft Teams).

[1312] The server sends the collected facial and voice data to the emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the emotional state of the attendees (joy, anger, sadness, surprise, etc.) in real time. For example, if an attendee's facial expression becomes stern and their voice tone drops, the emotion analysis engine will determine that the attendee is experiencing negative emotions. Based on this information, the server pauses the meeting and suggests a break to the attendees.

[1313] Furthermore, the emotion engine predicts changes in attendees' emotions based on past data. For example, if it is predicted that an attendee is likely to be dissatisfied with the meeting's progress, the server will provide that attendee with an opportunity to speak beforehand. It will also take actions such as adjusting the meeting's progress.

[1314] The following scenarios are possible as specific examples:

[1315] Scenario: During the meeting, participant A begins to express dissatisfaction.

[1316] Processing: The emotion analysis engine analyzes participant A's facial expressions and tone of voice and detects dissatisfaction. The server pauses the meeting and gives participant A an opportunity to speak.

[1317] Examples of prompts to input into a generative AI model:

[1318] "Design a system that analyzes attendees' emotions in real time during a meeting and, if negative emotions are detected, pauses the meeting and suggests a break. Also, add a function to predict changes in attendees' emotions and optimize the meeting's progress."

[1319] In this way, the server analyzes the emotions of the attendees and automatically performs appropriate facilitation, thereby realizing an environment that provides smooth communication even in a non-face-to-face setting. The flow of the specific processing in Example 3 will be explained using Figure 21.

[1320] Step 1:

[1321] The server activates the camera and microphone connected to the terminal as soon as the meeting starts. The terminal captures the attendees' facial expressions and audio data in real time and sends it to the server. The input is the video and audio data from the camera and microphone, and the output is the raw data sent to the server.

[1322] Step 2:

[1323] The server sends the received facial and audio data to the emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the emotional state of the attendees. The input is the raw data sent from the server, and the output is the evaluation result of the emotional state (joy, anger, sadness, surprise, etc.). Specifically, the emotion analysis engine analyzes facial features (e.g., eyebrow movement, mouth shape) and voice tone.

[1324] Step 3:

[1325] The server receives the emotional state evaluation results returned from the emotion analysis engine and determines whether the attendees' emotions are positive or negative. The input is the evaluation results from the emotion analysis engine, and the output is the judgment result of the emotional state. Specifically, if the server determines, based on the evaluation results, that the attendees' emotions are negative, it will temporarily suspend the meeting.

[1326] Step 4:

[1327] The server instructs the terminal to display a message suggesting a break if negative emotions are detected. The input is the result of the emotional state assessment, and the output is the break suggestion message displayed on the terminal. Specifically, the server sends a message to the terminal such as, "The attendee's emotions are leaning towards negative, so we suggest a 5-minute break."

[1328] Step 5:

[1329] The server predicts changes in attendees' emotions based on past emotional data. The input is past emotional data, and the output is the predicted result of those changes. Specifically, the emotion engine uses a machine learning algorithm to predict whether a particular attendee is likely to be dissatisfied with the progress of the meeting.

[1330] Step 6:

[1331] The server takes actions to optimize the meeting's progress based on predictions of emotional changes. The input is the predicted emotional changes, and the output is the optimized meeting progress instructions. Specifically, the server provides attendees who may have predicted dissatisfaction with an opportunity to speak in advance, and also takes actions such as adjusting the meeting's progress.

[1332] In this way, the server analyzes the emotions of the attendees and automatically performs appropriate facilitation, creating an environment that provides smooth communication even when not face-to-face.

[1333] (Application Example 3)

[1334] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1335] Traditional video conferencing and customer service systems have struggled to accurately analyze the emotions of attendees and customers and adjust responses in real time based on that analysis. This has resulted in problems such as meetings not running smoothly, decreased participant satisfaction, and inadequate customer service leading to low customer satisfaction. There is a need to address these issues and improve the quality of meetings and customer service.

[1336] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means.

[1337] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing customers' facial expressions and tone of voice to determine their emotions in real time, means for proposing appropriate responses based on the emotion analysis results, and means installed on smart devices to support customer service. This makes it possible to conduct meetings smoothly, improve the overall satisfaction of participants, and enable prompt and appropriate customer service, thereby improving customer satisfaction.

[1338] "Video conference facilitation" refers to the coordination and management required to ensure the smooth running of a video conference.

[1339] "Means of AI automation" refers to methods or devices that enable artificial intelligence to automatically perform specific tasks.

[1340] "Methods for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to methods or devices for analyzing the statements, facial expressions, and tone of voice of meeting participants and expressing their emotions as numerical values.

[1341] "Means of optimal meeting facilitation" refers to methods and devices for optimizing the progress of a meeting.

[1342] "Means of analyzing a customer's facial expressions and tone of voice to determine their emotions in real time" refers to methods or devices for analyzing a customer's facial expressions and tone of voice to determine their emotions in real time.

[1343] "Means for proposing appropriate responses based on emotion analysis results" refers to methods or devices for proposing appropriate responses based on the results of emotion analysis.

[1344] "Means installed on smart devices to support customer service" refers to methods and devices installed on devices such as smartphones and smart glasses to support customer service.

[1345] This invention is a system that uses AI to analyze emotions during video conferences and customer service interactions, and proposes appropriate responses in real time. The system includes functions for automatically facilitating smooth video conference proceedings and functions to support customer service interactions.

[1346] System Configuration

[1347] This system consists of the following main components:

[1348] 1. Video conferencing facilitation module: AI automatically manages the progress of video conferences.

[1349] 2. Emotion Analysis Module: Analyzes the content of attendees' and customers' statements, facial expressions, and tone of voice to quantify their emotions.

[1350] 3. Response Proposal Module: Based on the emotion analysis results, it proposes the optimal meeting facilitation and customer response.

[1351] 4. Smart Device Interface: Installed on devices such as smartphones and smart glasses to support customer service.

[1352] Hardware and software to be used

[1353] Hardware: Smart glasses, smartphone, camera, microphone

[1354] Software: OpenCV (face detection), Keras (sentiment analysis model)

[1355] Data processing and data calculation

[1356] 1. Face Detection: Use smart glasses or smartphone cameras to capture the faces of customers and attendees in real time. OpenCV is used to detect faces and acquire facial expression data.

[1357] 2. Emotion Analysis: The acquired facial expression data is input into Keras's emotion analysis model to quantify emotions. This allows for real-time assessment of the emotional state of customers and attendees.

[1358] 3. Suggested Response: Based on the emotion analysis results, appropriate responses are suggested. For example, if a customer is dissatisfied, the smart glasses display will show "The customer is dissatisfied. Please improve your response."

[1359] Specific example

[1360] scenario

[1361] When a store employee interacts with a customer and the customer shows signs of dissatisfaction, the smart glasses display will show the message, "The customer is dissatisfied. Please improve your service." This allows the employee to quickly improve their service and increase customer satisfaction.

[1362] Example of a prompt

[1363] "Capture the customer's facial expressions and use an emotion analysis model to determine their emotions. If the customer is dissatisfied, display a message suggesting improvements to the service."

[1364] In this way, a system that supports video conferencing and customer service in physical stores can be realized. This system makes it possible to conduct meetings smoothly, improve the overall satisfaction of participants, and enable quick and appropriate customer service, thereby improving customer satisfaction.

[1365] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[1366] Step 1:

[1367] Face detection

[1368] The server captures the faces of customers and attendees in real time using smart glasses or smartphone cameras. It receives video data from the cameras as input and detects faces using OpenCV. The output includes the location information and facial expression data of the detected faces. Specifically, it analyzes the camera footage frame by frame to identify the contours of the faces.

[1369] Step 2:

[1370] Emotion analysis

[1371] The server inputs the facial expression data obtained in Step 1 into the Keras emotion analysis model. It receives facial position information and facial expression data as input, and uses the emotion analysis model to quantify emotions. The output is the emotional state of the customer or attendee (e.g., joy, sadness, anger). Specifically, it preprocesses the facial expression data and inputs it into the emotion analysis model to predict emotions.

[1372] Step 3:

[1373] Proposed solutions

[1374] The server proposes an appropriate response based on the emotion analysis results obtained in step 2. It receives the emotional state as input and determines the appropriate response using the response suggestion module. As output, it generates a message to be displayed on smart glasses or a smartphone screen. Specifically, it searches the database for a response corresponding to the emotional state and generates a response message.

[1375] Step 4:

[1376] Message display

[1377] The device displays the corresponding message generated in step 3 on the smart glasses or smartphone display. It receives the corresponding message as input and processes it for display. As output, it visually presents the message to the user. Specifically, it calls the display's API to display the message.

[1378] Step 5:

[1379] User response

[1380] Based on the message displayed in Step 4, the user takes appropriate action towards customers and attendees. The input is receiving the displayed message and implementing the appropriate response. The output is improving customer and attendee satisfaction. Specifically, the user adjusts customer interactions and meeting progress according to the message.

[1381] In this way, the system can analyze emotions in real time and suggest appropriate responses, thereby improving the quality of meetings and customer interactions.

[1382] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1383] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1385] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1386] [Third Embodiment]

[1387] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1388] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1389] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1390] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1391] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1393] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1394] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1395] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1396] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1397] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1398] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[1399] "Example of form 1"

[1400] One embodiment of the present invention is a system in which AI automatically facilitates video conferences. Specifically, it is equipped with an AI engine that analyzes the content of what attendees say, their facial expressions, and their tone of voice. This AI engine uses speech recognition technology and image recognition technology to quantify the emotions of the attendees and performs optimal conference facilitation based on the results.

[1401] "Example of form 2"

[1402] Furthermore, it also features functions that provide specialized facilitation, such as patent-focused or strategy-focused facilitation. For example, in patent-focused facilitation, AI automatically asks questions and makes suggestions to facilitate discussions related to patents. This reduces the burden on users and improves the quality and effectiveness of meetings.

[1403] "Example of form 3"

[1404] Furthermore, it also has a function that provides an environment for smooth communication even when not face-to-face. Specifically, the AI ​​analyzes the emotions of the attendees and adjusts the progress of the meeting based on the results. For example, if it determines that the attendees' emotions are leaning towards negative, the AI ​​can automatically perform appropriate facilitation, such as pausing the meeting and suggesting a break to the attendees, just as a human would.

[1405] The following describes the processing flow for each example of the form.

[1406] "Example of form 1"

[1407] Step 1: The AI ​​engine starts the video conference and analyzes the participants' statements, facial expressions, and tone of voice in real time.

[1408] Step 2: The AI ​​engine uses speech recognition and image recognition technologies to quantify the emotions of the attendees.

[1409] Step 3: The AI ​​engine performs optimal meeting facilitation based on the analysis results. For example, if it determines that the attendees' emotions are leaning towards negative, it will take action such as pausing the meeting and suggesting a break to the attendees.

[1410] "Example of form 2"

[1411] Step 1: The AI ​​engine provides specialized facilitation, such as patent-focused or strategy-focused services.

[1412] Step 2: In patent-focused facilitation, AI automatically asks questions and makes suggestions to facilitate discussions related to patents.

[1413] Step 3: This reduces the burden on users and improves the quality and effectiveness of meetings.

[1414] "Example of form 3"

[1415] Step 1: Provide an environment where the AI ​​engine can communicate smoothly even without face-to-face interaction.

[1416] Step 2: The AI ​​analyzes the emotions of the attendees and adjusts the meeting's progress based on the results.

[1417] Step 3: For example, if the AI ​​determines that the attendees' emotions are leaning towards the negative, it can automatically perform appropriate facilitation actions similar to those a human would, such as pausing the meeting and suggesting a break to the attendees.

[1418] (Example 1)

[1419] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1420] Traditional video conferencing systems struggled to properly analyze participants' speech, facial expressions, and tone of voice to optimize meeting progress. Furthermore, the lack of means to provide feedback to improve meeting quality and effectiveness resulted in a significant burden on users and hindered smooth non-face-to-face communication.

[1421] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1422] In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice and quantifying their emotions, means for converting audio data into text using speech recognition technology, means for analyzing facial expressions from video data using image recognition technology, means for performing optimal conference facilitation based on the quantified emotion data, and means for providing feedback based on the progress of the conference and the emotion data. This makes it possible to analyze the emotions of attendees in real time and achieve optimal conference progress. Furthermore, by providing feedback to improve the quality and effectiveness of the conference, the burden on users is reduced and smooth communication can be achieved even without face-to-face interaction.

[1423] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[1424] "Facilitation" refers to techniques used to facilitate meetings and group discussions, enabling participants to effectively exchange opinions.

[1425] "Artificial intelligence" refers to a computer system that mimics human intelligence and possesses functions such as learning, reasoning, and recognition.

[1426] "Statements" refer to the words and opinions that participants express orally during a meeting.

[1427] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[1428] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and intonation, and is an element that conveys the speaker's emotions and intentions.

[1429] "Analysis" is the process of examining data in detail to understand its structure and meaning.

[1430] "Quantifying emotions" means quantitatively evaluating emotions and expressing them as numerical values.

[1431] "Speech recognition technology" is a technology that converts speech into text, allowing the speaker's words to be automatically recorded as text.

[1432] "Image recognition technology" is a technique that analyzes image data and recognizes specific patterns or features.

[1433] "Optimal meeting facilitation" refers to techniques for conducting a meeting most effectively, taking into account the circumstances and emotions of the participants.

[1434] "Feedback" refers to evaluations and advice provided based on the progress and results of a meeting.

[1435] "Progress status" refers to how the meeting is progressing, indicating its current progress and status.

[1436] "Emotional data" refers to data that quantifies the emotions of participants, and can be used to facilitate and manage meetings.

[1437] This invention relates to a system that uses artificial intelligence to automatically facilitate video conferences. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to achieve optimal conference facilitation.

[1438] Hardware and software to be used

[1439] Hardware: Camera, microphone, device (PC, tablet, smartphone)

[1440] Software: Speech recognition technology (e.g., Google Speech-to-Text), image recognition technology (e.g., OpenCV), AI engine (e.g., TensorFlow)

[1441] System operation

[1442] Subject: Server

[1443] The server provides a system that automatically facilitates video conferences. This system is equipped with an AI engine that analyzes the content of participants' speech, facial expressions, and tone of voice. The server uses speech recognition and image recognition technologies to quantify the emotions of participants. Based on this quantified emotion data, the server performs optimal meeting facilitation. It also provides feedback based on the progress of the meeting and the emotion data.

[1444] Subject: terminal

[1445] The device transmits video and audio from the video conference to the server. The device uses its camera and microphone to capture participants' facial expressions and speech. This data is sent to the server in real time and analyzed by an AI engine.

[1446] Subject: User

[1447] Users participate in video conferences and send their speech and facial expressions to the server via their devices. Users simply participate in the meeting naturally without performing any special operations. The content of the user's speech, facial expressions, and tone of voice are analyzed by an AI engine to provide optimal meeting facilitation.

[1448] Specific example

[1449] Example 1: Analysis of speech during a meeting

[1450] When User A says, "What is the progress of this project?", the server uses speech recognition technology to convert the statement into text. The AI ​​engine then analyzes this text and recognizes that User A is asking a question. The server then facilitates the discussion, prompting other attendees to provide appropriate answers.

[1451] Example 2: Facial expression analysis

[1452] The terminal's camera captures user B's facial expression and sends it to the server. The server uses image recognition technology to analyze user B's facial expression and detects that user B is confused. The server pauses the meeting and facilitates the discussion by asking user B questions or comments.

[1453] Example of a prompt

[1454] Please describe a system that analyzes participants' speech, facial expressions, and tone of voice during video conferences to provide optimal meeting facilitation. Please include specific hardware and software names.

[1455] In this way, the system's operation can be explained from the perspectives of the server, terminal, and user, and understanding can be deepened with concrete examples. This system makes it possible to analyze the emotions of attendees in real time and achieve optimal meeting management. Furthermore, by providing feedback to improve the quality and effectiveness of meetings, it reduces the burden on users and enables smooth communication even in remote settings.

[1456] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1457] Step 1: Start the video conference

[1458] Subject: User

[1459] The user starts a video conference using their device. The user launches the meeting application, enters the meeting ID, and joins the meeting. For example, if a user joins a meeting using a meeting app, they enter the meeting ID and click the "Join" button.

[1460] Input: Meeting ID, User actions

[1461] Output: Start video conference, prepare video and audio capture.

[1462] Step 2: Capture

[1463] Subject: terminal

[1464] The device uses a camera and microphone to capture the user's video and audio. The camera captures the user's facial expressions in real time, and the microphone records what the user says. For example, the device's camera captures the user's smile, and the microphone records them saying "hello."

[1465] Input: User video and audio

[1466] Output: Captured video and audio data

[1467] Step 3: Data transmission

[1468] Subject: terminal

[1469] The device sends the captured video and audio data to the server. The data is encrypted and sent securely to the server. For example, the device sends the captured video and audio data to the server using the HTTPS protocol.

[1470] Input: Captured video and audio data

[1471] Output: Data sent to the server

[1472] Step 4: Data Analysis

[1473] Subject: Server

[1474] The server analyzes the received video and audio data. It uses speech recognition technology to convert the audio data into text and image recognition technology to analyze facial expressions from the video data. For example, the server uses speech recognition technology to convert the audio "hello" into text and image recognition technology to detect the user's smile.

[1475] Input: Video and audio data sent to the server

[1476] Output: Text data, facial expression analysis results

[1477] Step 5: Quantifying emotions

[1478] Subject: Server

[1479] The server quantifies the user's emotions based on the analysis results. The AI ​​engine analyzes changes in voice tone and facial expressions to generate an emotion score. For example, the server might quantify the user's smile as "joy" and assign an emotion score of 80 / 100.

[1480] Input: Text data, facial expression analysis results

[1481] Output: Sentiment score

[1482] Step 6: Facilitating

[1483] Subject: Server

[1484] The server uses quantified sentiment data to provide optimal meeting facilitation. For example, if a user appears confused, the server pauses the meeting and prompts the user to ask questions or offer opinions.

[1485] Input: Sentiment score

[1486] Output: Facilitation instructions

[1487] Step 7: Provide feedback

[1488] Subject: Server

[1489] The server provides feedback based on the progress of the meeting and sentiment data. For example, after the meeting ends, it sends attendees a report about changes in their sentiment and the frequency of their comments during the meeting.

[1490] Input: Meeting progress, sentiment data

[1491] Output: Feedback Report

[1492] By specifically describing each processing step and clearly indicating the inputs and outputs, the system's operation can be understood more clearly.

[1493] (Application Example 1)

[1494] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1495] Conventional video conferencing systems have struggled to analyze participants' speech, facial expressions, and tone of voice to quantify their emotions and facilitate meetings effectively. Furthermore, while efficient meetings and discussions within factories require efficient progress, there was a lack of automated methods to manage meetings while considering participants' emotions. This resulted in decreased meeting quality and effectiveness, and increased burden on users.

[1496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1497] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for installation in a robot for streamlining meetings and discussions within a factory, and means for optimizing meeting progress while considering the emotions of attendees. This makes it possible to conduct efficient meetings and discussions within a factory while considering the emotions of attendees.

[1498] A "video conference" is a meeting conducted via the internet, where multiple participants share video and audio in real time.

[1499] "Facilitation" refers to the methods and techniques used to ensure that meetings and discussions proceed smoothly.

[1500] "AI" is an abbreviation for artificial intelligence, which is a technology that allows computers to learn and reason by mimicking human intelligence.

[1501] "Statements" refer to the information and opinions that participants express orally during meetings or discussions.

[1502] "Facial expression" refers to the emotions and reactions shown through the movement of the facial muscles.

[1503] "Voice quality" refers to the tone and texture of a voice, and is an element that reflects the speaker's emotions and intentions.

[1504] "Quantifying emotions" means analyzing the emotional state of participants and expressing it as numerical data.

[1505] "Optimal meeting facilitation" means taking into account the participants' comments and emotional states to conduct the meeting in the most effective way possible.

[1506] A "factory meeting" refers to a meeting held to discuss matters related to the operation and production of the factory.

[1507] A "meeting" is when several people gather to exchange opinions and share information regarding a specific purpose or issue.

[1508] A "robot" is a mechanical device that operates automatically according to a program and performs specific tasks.

[1509] "Installation" refers to bringing software or applications into a computer or machine and making them usable.

[1510] "Optimizing meeting progress" means making the necessary adjustments and improvements to ensure that meetings are conducted efficiently and effectively.

[1511] As an embodiment of this invention, we describe a "smart meeting robot" system for streamlining meetings and discussions within a factory. This system uses AI to automatically facilitate video conferences, analyzing the content of participants' statements, facial expressions, and tone of voice to quantify their emotions and perform optimal meeting facilitation.

[1512] System Configuration

[1513] 1. Hardware:

[1514] Camera: Used to capture attendees' facial expressions in real time.

[1515] Microphone: Used to collect the content of what attendees say.

[1516] Robot: A mechanical device that moves around within a factory and supports the progress of meetings.

[1517] 2. Software:

[1518] Speech recognition library: The speech_recognition library is used to recognize attendees' speech in real time.

[1519] Emotion analysis model: Using the transformers library's pipeline, the model analyzes the facial expressions and tone of voice of attendees and quantifies their emotions.

[1520] Image processing library: The cv2 library is used to acquire video from the camera in real time and perform facial expression analysis.

[1521] System operation

[1522] 1. Speech recognition:

[1523] The server converts the audio data acquired from the microphone into text data using the speech_recognition library.

[1524] The converted text data will be used as information to optimize the meeting's progress.

[1525] 2. Emotion analysis:

[1526] The server processes the video data acquired from the camera in real time using the cv2 library and analyzes the facial expressions of the attendees.

[1527] The analyzed facial expression data is then converted into numerical emotion using the transformers library's pipeline.

[1528] 3. Optimizing meeting facilitation:

[1529] The server executes logic to optimize the meeting's progress based on the acquired audio and emotion data.

[1530] For example, if an attendee says, "There's a problem with this part," the robot will respond, "A problem has occurred. I will suggest a solution."

[1531] Additionally, if an attendee's facial expression is negative, the message "Attendees are feeling negative. We will adjust the meeting's progress." will be displayed.

[1532] Specific example

[1533] As a concrete example, let's consider a quality control meeting in a factory. If an attendee says, "There is a problem with this part," the robot will respond, "A problem has occurred. I will propose a solution." Also, if the attendee's expression is negative, it will display, "The attendee's mood is negative. I will adjust the meeting's progress."

[1534] Example of a prompt

[1535] Examples of prompt statements to input into a generative AI model include the following:

[1536] If someone says, "There's a problem with this part," please suggest how the meeting should proceed.

[1537] In this way, the smart meeting robot system can streamline meetings within factories and provide optimal facilitation that takes into account the emotions of the attendees.

[1538] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1539] Step 1:

[1540] The server acquires video data from the camera in real time. The input is video data from the camera, and the output is frame data processed using the image processing library cv2. Specifically, the server captures the video from the camera frame by frame, converts it to grayscale, and performs face detection.

[1541] Step 2:

[1542] The server acquires audio data from the microphone in real time. The input is audio data from the microphone, and the output is the spoken content converted into text data using the speech recognition library speech_recognition. Specifically, the server records the audio from the microphone and converts it to text using the speech recognition engine.

[1543] Step 3:

[1544] The server analyzes the facial expressions of attendees using the acquired frame data. The input is the frame data obtained in step 1, and the output is emotion data quantified using the pipeline of the emotion analysis model transformers. Specifically, the server extracts the region where a face was detected and inputs it into the emotion analysis model to quantify the emotion.

[1545] Step 4:

[1546] The server analyzes the acquired text data and extracts information necessary for the meeting's progress. The input is the text data obtained in step 2, and the output is instructions and suggestions for the meeting's progress. Specifically, the server inputs the text data into a natural language processing engine and extracts important keywords and phrases.

[1547] Step 5:

[1548] The server executes logic to optimize the meeting's progress based on sentiment data and text data. The inputs are the sentiment data obtained in step 3 and the text data obtained in step 4, and the output is specific actions for managing the meeting. Specifically, the server integrates the sentiment data and text data and generates instructions to adjust the meeting's progress.

[1549] Step 6:

[1550] The server sends the generated instructions to the robot to support the progress of the meeting. The input is the meeting progress instructions obtained in step 5, and the output is the specific actions taken by the robot. Specifically, the server sends voice and motion instructions to the robot to support the progress of the meeting.

[1551] Step 7:

[1552] The user conducts the meeting by following instructions from the robot. The input is the instructions from the robot, and the output is the progress of the meeting. Specifically, the user speaks and acts according to the robot's instructions to ensure the meeting runs smoothly.

[1553] (Example 2)

[1554] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1555] Traditional video conferencing systems often require manual facilitation, placing a heavy burden on attendees and potentially reducing the quality and effectiveness of meetings. Furthermore, discussions related to patents require specialized knowledge, making it difficult to generate appropriate questions and suggestions. Additionally, non-face-to-face communication is often inefficient, leading to delays in meeting progress.

[1556] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1557] In this invention, the server includes means for artificial intelligence to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice and quantifying their emotions, means for optimal conference facilitation, means for collecting patent-related data, means for pre-processing the collected data, means for generating questions and suggestions using a generative AI model, means for transmitting the generated questions and suggestions to a terminal, and means for the terminal to present the results to the user. This reduces the burden on the user and improves the quality and effectiveness of meetings. It also facilitates specialized discussions related to patents and enables smooth communication even in non-face-to-face settings.

[1558] A "video conference" is a type of meeting in which multiple participants communicate in real time using video and audio over the internet.

[1559] "Facilitation" refers to activities that provide support and coordination to ensure the smooth progress of meetings and discussions.

[1560] "Artificial intelligence" is a technology in which computer systems imitate human intelligence to learn, reason, and self-correct.

[1561] "Statements" refer to the information and opinions that participants express orally during a meeting or discussion.

[1562] "Facial expression" refers to emotions and reactions expressed through the movement of facial muscles.

[1563] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and pitch.

[1564] "Quantifying emotions" means expressing emotional states as numerical data.

[1565] A "patent" is a legal right that grants an inventor the exclusive right to use their invention for a certain period of time.

[1566] "Collecting data" is the act of gathering necessary information.

[1567] "Preprocessing" refers to the initial processing required to prepare data into a format that is easy to analyze and use.

[1568] A "generative AI model" is a model that uses artificial intelligence to generate new data and information.

[1569] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific theme.

[1570] A "terminal" is a device used by a user to input or output information.

[1571] A "user" is a person who uses a system or service.

[1572] "Presenting results" means showing the generated information or data to the user.

[1573] This invention is a system that uses artificial intelligence to automatically facilitate video conferences. The system has the function of analyzing the content of participants' speech, facial expressions, and tone of voice, and quantifying their emotions. It also includes a function to collect patent-related data, perform preprocessing, and generate questions and suggestions using a generative AI model. A specific embodiment of this system is described below.

[1574] Hardware and software configuration

[1575] server

[1576] The server is equipped with the hardware and software necessary to perform the following functions:

[1577] Data collection: The server collects the necessary information from patent-related databases (e.g., patent information databases).

[1578] Data preprocessing: The server uses Python's NLTK library and spaCy to preprocess the collected data.

[1579] Application of Generative AI Models: The server uses a pre-trained generative AI model (e.g., GPT-4) to generate patent-related questions and suggestions.

[1580] Sending results: The server sends the generated questions and suggestions to the terminal.

[1581] terminal

[1582] A terminal is a device used by users to input information and display results sent from a server. A terminal has the following functions:

[1583] Receiving user input: The terminal receives prompts entered by the user.

[1584] Displaying results: The terminal presents the user with questions and suggestions sent from the server.

[1585] User

[1586] A user is someone who uses the system to conduct a video conference. The user performs the following actions:

[1587] Prompt input: The user enters prompts into the terminal to facilitate discussions related to the patent.

[1588] Specific example

[1589] For example, consider a scenario where a user is conducting a meeting about patents.

[1590] Example of a prompt:

[1591] "We would like to discuss a new patent application. Please generate questions and suggestions regarding the patent's technical details and the competitive landscape of the market."

[1592] When the user enters this prompt into the terminal, the server collects relevant data from the patent information database and performs preprocessing. Next, it uses a generative AI model to generate appropriate questions and suggestions and sends them to the terminal. The terminal then presents the generated questions and suggestions to the user. For example, a question such as "How does this technology differ from existing patents?" might be displayed.

[1593] In this way, users can use the system to efficiently advance discussions related to patents. The system can reduce the burden on users and improve the quality and effectiveness of meetings. It can also enable smooth communication even when meetings are not face-to-face.

[1594] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1595] Step 1:

[1596] The user enters a prompt message.

[1597] The user enters prompts into the terminal to facilitate discussions related to the patent. For example, they might enter, "I would like to discuss a new patent application. Please generate questions and suggestions regarding the patent's technical details and market competitive landscape." The entered prompts are then sent from the terminal to the server.

[1598] Step 2:

[1599] The server collects the data.

[1600] The server receives prompt messages from the user and collects patent-related data. Specifically, it retrieves relevant patent information from a patent information database. For example, it calls the API of the patent information database to collect patent information related to "new technologies." The input is the prompt message, and the output is the collected patent information.

[1601] Step 3:

[1602] The server preprocesses the data.

[1603] The server preprocesses the collected patent information. Using Python's NLTK library and spaCy, it analyzes the text data and extracts important keywords and phrases. For example, it extracts keywords such as "technical details" and "market competitive landscape." The input is the collected patent information, and the output is the preprocessed data.

[1604] Step 4:

[1605] The server applies the generated AI model.

[1606] The server generates questions and suggestions using a generative AI model (e.g., GPT-4) based on pre-processed data. The generative AI model generates appropriate questions and suggestions based on extracted keywords. For example, it might generate a question such as, "How does this technology differ from existing patents?" The input is pre-processed data, and the output is the generated questions and suggestions.

[1607] Step 5:

[1608] The server sends the generated results to the terminal.

[1609] The server sends the generated questions and suggestions to the terminal. The generated questions and suggestions are sent to the terminal in JSON format. For example, they are sent in the format {"question": "How does this technology differ from existing patents?"}. The input is the generated questions and suggestions, and the output is the data sent to the terminal.

[1610] Step 6:

[1611] The device presents the results to the user.

[1612] The terminal presents the user with questions and suggestions received from the server. The user can then proceed with the discussion based on the presented questions and suggestions. For example, the terminal screen might display "How does this technology differ from existing patents?". The input is data sent from the server, and the output is the questions and suggestions presented to the user.

[1613] (Application Example 2)

[1614] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1615] Traditional video conferencing systems require manual facilitation, which places a heavy burden on attendees and can lead to a decline in the quality and effectiveness of meetings. Furthermore, discussions related to patents and technical debates require specialized knowledge, making it difficult to ask appropriate questions or make suggestions. Additionally, the inability to facilitate smooth non-face-to-face communication was a problem.

[1616] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1617] In this invention, the server includes means for an AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for generating questions and suggestions to promote discussions related to patents, and means for generating prompt sentences based on the meeting topic using a generative AI model. This improves the quality and effectiveness of meetings, efficiently supports discussions and technical debates related to patents, and enables smooth communication even when not face-to-face.

[1618] A "video conference" is a meeting conducted via the internet, in which multiple participants share video and audio in real time.

[1619] "Facilitation" refers to activities that provide support and coordination to ensure that meetings and discussions proceed smoothly.

[1620] "AI" is an abbreviation for artificial intelligence, a technology in which computers imitate human intelligence to perform learning and reasoning.

[1621] "Statements" refer to the information and opinions expressed orally by participants during meetings or discussions.

[1622] "Facial expression" refers to emotions and reactions expressed through the movement of facial muscles.

[1623] "Voice quality" refers to the characteristics of a voice, such as its sound quality, tone, and intonation.

[1624] "Quantifying emotions" means expressing emotional states as numerical data.

[1625] A "patent" is an exclusive right granted to an invention.

[1626] "To facilitate discussion" means to support participants in actively exchanging opinions and engaging in lively debate.

[1627] "Generating questions and suggestions" means automatically creating appropriate questions and suggestions based on a specific topic.

[1628] A "generative AI model" is an artificial intelligence model that generates new data or information based on input data.

[1629] A "prompt statement" refers to an instruction or question that is input into a generative AI model.

[1630] "Factory interior" refers to the inside of a facility where production activities take place in the manufacturing industry.

[1631] A "technical discussion" is a meeting where experts exchange opinions and debate technical matters.

[1632] A system for carrying out this invention includes a program for automatically facilitating video conferences. The system is implemented using the following hardware and software.

[1633] hardware

[1634] Server: A server with high-performance computing capabilities is required. This will enable rapid execution of generative AI models and data analysis.

[1635] Device: You will need a device (such as a PC, tablet, or smartphone) to participate in the video conference.

[1636] Cameras and microphones: Cameras and microphones are necessary to capture what attendees say, their facial expressions, and their tone of voice.

[1637] software

[1638] Generative AI Model: Uses the OpenAI API to generate prompt sentences based on the meeting topic.

[1639] Data analysis software: Software is needed to analyze the content of attendees' statements, facial expressions, and tone of voice, and to quantify their emotions.

[1640] Video conferencing software: You will need software to conduct video conferences (for example, Zoom or Microsoft Teams).

[1641] Data processing and data calculation

[1642] The server analyzes the participants' statements, facial expressions, and tone of voice captured during video conferences in real time, quantifying their emotions. This allows for understanding the progress of the meeting and the participants' reactions. Furthermore, using a generative AI model, it generates prompt sentences based on the meeting topic and automatically asks questions and makes suggestions to facilitate discussions related to patents.

[1643] Specific example

[1644] For example, when conducting a meeting about a patent for a new manufacturing process, the server generates a prompt message like the following:

[1645] "What technical features should we emphasize to obtain a patent for this manufacturing process?"

[1646] "What are the advantages of this process compared to the patents of our competitors?"

[1647] This allows attendees to conduct discussions more efficiently, improving the quality and effectiveness of meetings. It also enables smooth communication even when not face-to-face.

[1648] In this way, the system for implementing the invention can automate the facilitation of video conferences and support patent-related discussions and technical debates.

[1649] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1650] Step 1:

[1651] The server activates the camera and microphone at the start of a video conference to capture participants' speech, facial expressions, and tone of voice. It receives real-time data from the camera and microphone as input and sends this data to analysis software as output.

[1652] Step 2:

[1653] The server analyzes the acquired speech content, facial expressions, and tone of voice data using analysis software to quantify emotions. It receives real-time data from the camera and microphone as input and generates quantified emotion data as output. Specifically, it uses speech recognition technology to transcribe speech into text and facial recognition technology to analyze facial muscle movements.

[1654] Step 3:

[1655] The server generates prompt sentences based on the meeting topic using a generative AI model, based on the analysis results. It receives quantified sentiment data and the meeting topic as input, and generates prompt sentences as output. Specifically, it uses the OpenAI API to generate the prompt sentences.

[1656] Step 4:

[1657] The server sends the generated prompt message to the video conferencing software and presents it to the attendees. It receives the generated prompt message as input and displays the prompt message in the video conferencing software as output. Specifically, it displays the prompt message using the chat or screen sharing functions of the video conferencing software.

[1658] Step 5:

[1659] The user advances the discussion based on the presented prompt. It receives the prompt as input and generates the discussion content as output. Specifically, the user provides opinions and questions in response to the prompt.

[1660] Step 6:

[1661] The server monitors the progress of the discussion and generates additional prompts as needed. It receives discussion content and sentiment data as input and generates additional prompts as output. Specifically, it uses the generative AI model again to generate new prompts.

[1662] Step 7:

[1663] The server saves all data and generates a meeting report at the end of the meeting. It receives all data acquired during the meeting as input and generates a meeting report as output. Specifically, it saves the data to a database and creates the report using report generation software.

[1664] (Example 3)

[1665] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1666] Traditional video conferencing systems struggled to analyze participants' emotions in real time and appropriately adjust the meeting's progress. This resulted in decreased meeting quality and effectiveness, and increased user burden. Furthermore, the inability to communicate effectively without face-to-face interaction often led to delays in meetings.

[1667] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for artificial intelligence to automatically facilitate the meeting, means for analyzing the content of attendees' statements, facial expressions, and tone of voice and quantifying their emotions, means for optimal meeting facilitation, means for collecting meeting audio data, means for transmitting the collected audio data to the server, means for the server to perform emotion analysis on the audio data, means for adjusting the progress of the meeting based on the analysis results, and means for suggesting appropriate actions to the user. This improves the quality and effectiveness of meetings, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1668] "Methods for using artificial intelligence to automatically facilitate meetings" refers to functions that use artificial intelligence to automatically manage and coordinate meetings.

[1669] "A means of analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions" refers to a function that analyzes the statements, facial expressions, and tone of voice of attendees and expresses those emotions as numerical values.

[1670] "A means of optimally facilitating meetings" refers to a function that makes adjustments to optimize the progress of meetings based on analyzed emotional data.

[1671] "Means for collecting meeting audio data" refers to a function for collecting audio during a meeting in real time.

[1672] "Means for sending collected audio data to a server" refers to a function for sending collected audio data to a server via the internet.

[1673] "Means for a server to analyze the emotions of audio data" refers to a function in which the server analyzes the received audio data to identify the emotions of the attendees.

[1674] "Means for adjusting the progress of a meeting based on analysis results" refers to a function that appropriately adjusts the progress of a meeting based on the results of sentiment analysis.

[1675] "Means of suggesting appropriate actions to users" refers to functions that suggest appropriate actions to users based on analysis results.

[1676] This invention relates to a system that uses artificial intelligence to automatically facilitate meetings. This system analyzes the content of participants' statements, facial expressions, and tone of voice, and quantifies their emotions to enable optimal meeting facilitation.

[1677] 1. Generate the system program.

[1678] This system's program is developed using Python. The main libraries used are "Hugging Face Transformers" for sentiment analysis and the "Google Calendar API" for meeting management.

[1679] 2. Explain the program's processing in natural language.

[1680] When a user starts a meeting, the device collects the meeting's audio data in real time. The collected audio data is sent to a server, where sentiment analysis is performed using Hugging Face Transformers. The analysis results are categorized into three categories: positive, negative, and neutral.

[1681] The server adjusts the meeting's progress based on the analysis results. For example, if it determines that an attendee's emotions are leaning towards negative, the server will pause the meeting via the Google Calendar API and display a message on the device suggesting a break.

[1682] 3. Add specific examples to the explanation.

[1683] The following scenario is a concrete example.

[1684] scenario:

[1685] A user starts an online meeting. During the meeting, one of the attendees repeatedly makes negative remarks. The system performs sentiment analysis on the remarks and determines that the negative emotions are strong. The server pauses the meeting and displays the message "Do you want to suggest taking a break?" on the user's device.

[1686] Example of a prompt:

[1687] "Develop a system that analyzes participants' emotions in real time during online meetings and suggests pausing the meeting and taking a break if negative emotions are strongly present."

[1688] This system provides an environment that enables smooth communication even without face-to-face interaction. The flow of a specific process in Example 3 will be explained using Figure 15.

[1689] Step 1:

[1690] The user initiates the meeting. The user launches the meeting application and clicks the "Start Meeting" button. This causes the system to begin collecting audio data. The input is the user's actions, and the output is the meeting start signal.

[1691] Step 2:

[1692] The device collects audio data from the meeting. The device uses a microphone to pick up audio during the meeting and converts it into digital audio data in real time. For example, if a user says "Hello everyone," that audio is immediately collected as digital data. The input is the audio from the meeting, and the output is digital audio data.

[1693] Step 3:

[1694] The terminal sends the collected audio data to the server. The terminal sends the collected audio data to the server using the HTTPS protocol. The data is AES encrypted and transmitted securely. The input is digital audio data, and the output is a signal indicating that transmission to the server is complete.

[1695] Step 4:

[1696] The server performs sentiment analysis on the audio data. The server inputs the received audio data into the Hugging Face Transformers sentiment analysis model. For example, the statement "This project might not work out" is analyzed as negative. The input is digital audio data, and the output is the sentiment analysis result.

[1697] Step 5:

[1698] The server adjusts the meeting's progress based on the analysis results. If the server determines that the sentiment analysis results are negative, it issues a command to temporarily suspend the meeting using the Google Calendar API. The input is the sentiment analysis result, and the output is a command to adjust the meeting's progress.

[1699] Step 6:

[1700] The terminal suggests appropriate actions to the user. The terminal receives instructions from the server and displays a message to the user saying, "Attendees are feeling negative. Would you like to suggest a break?" The user adjusts the meeting's progress by selecting "Yes" or "No." The input is the instruction from the server, and the output is the suggestion message to the user.

[1701] In this way, the system provides an environment where smooth communication can take place even without face-to-face interaction.

[1702] (Application Example 3)

[1703] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1704] Traditional video conferencing systems and work environments have struggled to adequately analyze the emotions of attendees and workers, thereby optimizing meeting progress and work efficiency. Furthermore, the inability to respond appropriately when emotions turned negative led to problems with reduced meeting quality and work safety. This resulted in increased user burden and hindered smooth non-face-to-face communication.

[1705] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means.

[1706] In this invention, the server includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' statements, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for temporarily suspending work and suggesting a break if emotions become negative. This optimizes the progress of meetings and work efficiency, reduces the burden on users, and enables smooth communication even in non-face-to-face settings.

[1707] A "video conference" is a meeting conducted over the internet, where multiple participants share video and audio.

[1708] "Facilitation" refers to providing support and coordination to ensure that meetings and work proceed smoothly.

[1709] "AI" stands for artificial intelligence, which is a technology in which machines imitate human intelligence to learn and reason.

[1710] "Statements" refer to the information and opinions that participants express orally during meetings or work sessions.

[1711] "Facial expression" refers to the emotions and intentions conveyed through the movement of the facial muscles.

[1712] "Voice quality" refers to the sound quality and tone of a voice, and it reflects the speaker's emotions and state of mind.

[1713] "Quantifying emotions" means expressing emotions as quantitative data.

[1714] A "worker" is a person who performs a specific task in a factory, office, or similar setting.

[1715] "Real-time" means processing events that are currently unfolding immediately.

[1716] "Work efficiency" refers to the ability to minimize wasted time and effort during work and achieve maximum results.

[1717] "Safety" refers to a state in which work or activities can be carried out without danger and with peace of mind.

[1718] "Suggesting a break" means temporarily interrupting work or a meeting and encouraging participants to take a rest.

[1719] A "system" is a mechanism in which multiple elements work together to perform a specific function.

[1720] The system for implementing this invention includes means for AI to automatically facilitate video conferences, means for analyzing the content of attendees' speech, facial expressions, and tone of voice to quantify their emotions, means for optimal meeting facilitation, means for analyzing workers' emotions in real time to improve work efficiency and safety, and means for temporarily suspending work and suggesting a break if emotions become negative.

[1721] System program

[1722] The program in this system performs the following operations:

[1723] 1. Hardware: The system uses a camera (e.g., Logitech C920) to capture the faces of attendees and workers.

[1724] 2. Software: The system uses software such as OpenCV, Keras, and TensorFlow to process captured images and analyze emotions.

[1725] 3. Data Processing: The captured images are converted to grayscale, and face detection is performed. The detected face regions are preprocessed for input into the emotion analysis model.

[1726] 4. Data processing: Use an emotion analysis model to predict emotions from pre-processed facial images.

[1727] Specific example of processing

[1728] For example, if a worker in a factory is experiencing stress, the system analyzes their emotions in real time, temporarily suspends their work, and suggests a break. This improves work efficiency and safety, and reduces the burden on workers.

[1729] Example of a prompt

[1730] Examples of prompts to input into a generative AI model are as follows:

[1731] Develop a robot assistant application that analyzes workers' emotions in real time within a factory to improve work efficiency and safety. Include a feature that allows the robot to pause work and suggest a break if a worker is experiencing stress.

[1732] In this way, the system optimizes meeting progress and work efficiency, reduces the burden on users, and enables smooth communication even when not face-to-face.

[1733] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[1734] Step 1:

[1735] The server uses a camera to capture the faces of attendees and workers. The input is video data from the camera, and the output is the captured image data. Specifically, the camera acquires video in real time, and the server captures that video frame by frame.

[1736] Step 2:

[1737] The server converts the captured image data to grayscale. The input is the captured image data, and the output is grayscale image data. Specifically, it uses OpenCV to convert the image to grayscale.

[1738] Step 3:

[1739] The server detects faces from grayscale images. The input is grayscale image data, and the output is the coordinate data of the detected face region. Specifically, it uses the OpenCV face detection algorithm to identify the face region.

[1740] Step 4:

[1741] The server preprocesses the detected face regions for input into the...

Claims

1. A method for using artificial intelligence to automatically facilitate video conferences, A means for converting the audio data of the video conference into text using speech recognition technology, A means for analyzing facial expressions from video data of the video conference using image recognition technology, A means for analyzing at least one of the participants' statements, facial expressions, and tone of voice in the aforementioned video conference and quantifying their emotions, The means for obtaining numerically represented emotion data using the aforementioned numerical means, and means for monitoring the progress of the video conference, which indicates the current progress or status of the video conference, A means for detecting changes in the emotions of the attendees in real time based on the aforementioned emotion data, A means for predicting changes in the emotions of the attendees based on the aforementioned emotional data and past emotional data, A means for optimizing the progress of a video conference and providing optimal meeting facilitation, by adjusting the progress of the meeting to temporarily suspend the meeting and suggest a break if it is determined that the emotions of the attendees are leaning towards negative, based on the results of detecting and predicting changes in emotions, and by providing an opportunity for the attendees to speak if it is determined that the attendees are showing anger or dissatisfaction. A means of providing feedback by generating and sending a feedback report to the attendees after the meeting, based on the progress of the meeting and emotional data, regarding changes in their emotions or the frequency of their comments during the meeting. A system that includes this.

2. A means for receiving input from the attendees regarding the theme or purpose of the video conference and collecting data related to said input from a database, A means for analyzing the collected data, extracting keywords or phrases, and preprocessing them; A means for generating questions or suggestions to facilitate expert discussion using a generative AI model based on the aforementioned preprocessed data, A means of transmitting the generated question or proposal to the attendee's terminal for presentation, The system according to claim 1, further comprising:

3. The means for performing the optimal meeting facilitation is: A means for integrating the aforementioned emotional data and the audio data of the video conference converted into text to generate instructions for adjusting the progress of the meeting, The system includes means for transmitting the generated instructions to the smart devices of attendees participating in the video conference or to a robot that supports the progress of the video conference, The system according to claim 1, wherein the smart device or robot has means for supporting the progress of the meeting by performing at least one of the following in accordance with the received instructions: outputting sound, displaying a message, and performing an action.

Images