System

Abstract

A system is provided.SOLUTION: A system includes a means for acquiring video data, a means for transmitting the acquired video data to a server, a means for analyzing the video data by the server, a means for generating a communication method on the basis of an analysis result, a means for presenting the generated communication method to a terminal, and a means for selecting the presented communication method by a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern times, remote meetings and online interviews are becoming more and more common, but compared to face-to-face meetings, there is a problem in that it is difficult to grasp non-verbal information obtained from the other person's facial expressions and gestures. This makes effective communication difficult, and there is a problem in that it is difficult to accurately grasp the other person's psychological state and intentions, especially in important dialogue situations such as interviews and counseling. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides a system including means for acquiring video data, means for transmitting the acquired video data to a server, means for the server to analyze the video data, means for generating a communication method based on the analysis results, means for presenting the generated communication method to a terminal, and means for a user to select the presented communication method.This system makes it possible to effectively obtain non-verbal information from the other party in a remote conference, just as in a face-to-face conference, and to achieve smooth communication.

[0006] "Video data" refers to visual information acquired using a camera or other imaging device, and specifically refers to data expressed in the form of images or videos.

[0007] "Acquiring" refers to the act of collecting video data using a camera or other imaging device.

[0008] "Send" refers to the act of transferring data from one device to another device or server.

[0009] "Analyzing" refers to the act of extracting information from acquired video data using specific algorithms or AI models and attempting to understand its meaning.

[0010] "Emotions" represent a person's psychological state, and are internal conditions that can be read from physical movements such as facial expressions and gestures.

[0011] A "server" is a computer system that processes and stores data over a network.

[0012] "Communication methods" refer to specific means or methods for effectively communicating and interacting with others.

[0013] "Presenting" refers to the act of visually displaying information or options to the user.

[0014] "User" refers to a person who uses this system to hold remote meetings or interviews.

[0015] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, gestures, eye contact, and body movements. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[0038] Device behavior

[0039] The terminal first starts operating when a user starts a remote conference. A camera is connected to the terminal, and video data of the user and the other party is acquired in real time using this camera. The acquired video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[0040] Server Operation

[0041] The server receives the video data sent from the device and instantly analyzes it using generative AI. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, and uses these to infer the other person's emotions and psychological state. Specifically, the generative AI identifies states such as happiness, tension, and concentration, and generates a communication method based on these. The analysis results are stored in a database and made available for immediate use.

[0042] Creation and presentation of communication methods

[0043] The server generates several effective communication methods based on the analysis results. The generated communication methods include multiple patterns, such as light-hearted conversations when the other person is relaxed, or reassuring words when the other person is nervous.

[0044] The generated communication methods are sent to the terminal and presented on the user's UI. The user can view the options and select the most suitable communication method. The user's choice from the displayed options is immediately applied, and the conversation continues.

[0045] Specific examples

[0046] For example, consider a case where a user is conducting a remote job interview. In this case, the device captures video data of the interviewer via a camera and sends it to a server. The server uses generative AI to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier." These options are presented to the user via the device, allowing them to choose the appropriate method and smoothly progress through the interview.

[0047] In this way, the present invention makes it possible to achieve effective communication in remote conferences that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[0048] The processing flow will be explained below.

[0049] Step 1:

[0050] A user starts a remote conference. The user uses a device (such as a PC or smartphone) to join the ZOOM conference using a camera and microphone. The user joins the conference URL on the ZOOM application or starts a new conference.

[0051] Step 2:

[0052] The device captures video data in real time using a camera, encodes the captured video data frame by frame, and temporarily stores it in a buffer. This video data is then sent to a server at regular intervals.

[0053] Step 3:

[0054] The server analyzes the received video data. The video data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. As a result of the analysis, the other person's emotions and psychological state are inferred.

[0055] Step 4:

[0056] The server generates several communication methods based on the analysis results. For example, if the person smiles a lot, it will judge them to be "relaxed," and if they have furrowed brows, it will judge them to be "tense," and generate a communication method appropriate to each.

[0057] Step 5:

[0058] The server generates a communication method and sends it to the device in real time, including multiple suggested patterns (e.g., humorous conversation, words of encouragement, etc.).

[0059] Step 6:

[0060] The device then presents the received communication method to the user. The user selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option. This selection allows for smooth communication that is close to face-to-face.

[0061] Step 7:

[0062] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[0063] In this way, the system of the present invention can effectively obtain non-verbal information in remote conferences similar to that in face-to-face conferences and provide appropriate means of communication, thereby improving the quality of dialogue.

[0064] Example 1

[0065] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0066] In remote conferences, non-face-to-face communication between participants poses the challenge of accurately predicting the other person's emotions and psychological state from their facial expressions and gestures, and using appropriate communication methods. As a result, the effectiveness of communication decreases, and the possibility of misunderstandings and breakdowns in dialogue increases.

[0067] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0068] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to a data processing device, and means for the data processing device to analyze the video data. This enables the user to analyze the emotions and psychological state of the other person and generate and present an effective communication method based on that analysis.

[0069] "Video data" refers to visual information captured using a camera or other optical device.

[0070] A "data processing device" is a device for analyzing video data and processing the information using specific algorithms.

[0071] "Analyzing" is the process of using captured video data to detect and evaluate specific patterns or characteristics.

[0072] "Communication methods" refer to the means and approaches of dialogue that are generated based on the emotions and psychological state of the other person.

[0073] A "display device" is a device for visually presenting the generated communication method to the user.

[0074] "User" refers to an individual or group that operates this system and interacts with it through a terminal.

[0075] A "means" is a device, process, or system element designed to accomplish a particular purpose.

[0076] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[0077] Device behavior

[0078] The device first begins operation when a user starts a remote conference. The specific hardware used is a camera (e.g., a standard webcam). This camera is used to capture video data of the user and the other party in real time. The captured video data is captured at a rate of 30 frames per second and compressed and encoded using FFmpeg software. The encoded video data is then sent to the server using WebSocket.

[0079] Server Operation

[0080] The server receives the video data sent from the device and immediately begins analysis. For analysis, a generative AI model (e.g., a general generative AI model and TensorFlow) is used. This generative AI extracts features such as facial expressions, gestures, complexion, and gaze from the video data, and infers psychological states such as happiness, tension, and concentration. The analysis results are stored in a database (e.g., a general relational database or NoSQL database).

[0081] Creation and presentation of communication methods

[0082] The server generates several effective communication methods based on the analysis results. This generation uses a generative AI model (e.g., a general generative AI model) with natural language generation (NLG) capabilities. The generated communication methods include multiple patterns, such as light-hearted banter when the other person is relaxed, or reassuring words when the other person is nervous. Below is an example of a prompt:

[0083] Sample prompt: "Generate appropriate communication suggestions if the other person is nervous."

[0084] The generated communication method is converted into JSON format and sent to the terminal via WebSocket communication.

[0085] Display on the device's user interface

[0086] The device presents the received communication method on a user interface (UI), for example, displaying options such as "Recommend deep breathing to relax" or "Start with a simple question to make it easier to talk."

[0087] User selection and application

[0088] The user selects the communication method they think is most appropriate from the options displayed on the device's UI. The selected method is immediately applied and the conversation continues.

[0089] Specific examples

[0090] For example, when a user conducts a remote job interview, the device uses a camera to capture video data of the interviewer and sends it to a server. The server uses a generative AI model to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier to understand," and sends these to the device. The device then presents these options on a user interface, allowing the user to select the appropriate method to smoothly proceed with the interview.

[0091] Through this procedure, the present invention enables effective communication in remote meetings that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Step 1:

[0094] A user starts the meeting software on a terminal and starts a remote conference. The input is the user's operation (a conference start action), and the output is the system entering an operational state.

[0095] Step 2:

[0096] The device uses a camera to capture real-time video data of the user and the target. The input is the video signal from the camera, and the output is the captured video data. FFmpeg is used to compress and encode the video data.

[0097] Step 3:

[0098] The device sends the encoded video data to the server using WebSocket communication. The input is the encoded video data, and the output is data packets via WebSocket.

[0099] Step 4:

[0100] The server receives video data sent from the device. The input is data packets sent over WebSocket, and the output is parseable video data, which the server stores in a buffer.

[0101] Step 5:

[0102] The server analyzes the video data using a generative AI model. The input is the received video data, and the output is characteristic data such as facial expressions, gestures, complexion, and gaze. Based on this data, the server processes the other person's emotions and psychological state.

[0103] Step 6:

[0104] The server generates several effective communication methods based on the analysis results. The input is the analysis results (emotions and psychological state), and the output is the generated multiple communication methods. A generative AI model with natural language generation (NLG) capabilities is used for generation.

[0105] Step 7:

[0106] The server converts the generated communication methods into JSON format and sends it to the terminal via WebSocket communication. The input is multiple communication methods (text data), and the output is JSON data, which is sent to the terminal via WebSocket.

[0107] Step 8:

[0108] The device parses the received JSON data and displays the generated communication methods on the user interface (UI). The input is JSON-formatted data, and the output is options on the UI. For example, options such as "Recommend deep breathing" or "Ask a simple question" are displayed.

[0109] Step 9:

[0110] The user selects the most appropriate communication method from the displayed options. The input is the UI option, and the output is the selected communication method. The selected method is applied immediately.

[0111] Step 10:

[0112] The device reports the selected communication method to the server and the dialogue continues. The input is the communication method selected by the user, and the output is the selection sent to the server, which uses this to prepare for the next analysis cycle.

[0113] (Application example 1)

[0114] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] In industrial settings, there is a need to grasp the emotions and psychological state of workers in real time and provide appropriate feedback accordingly. However, current systems have difficulty in doing so effectively, which can lead to a decline in worker efficiency and performance. In addition, there is a lack of means to provide instructions tailored to specific situations in real time, resulting in a lack of immediate response capabilities on-site. This poses challenges for improving worker safety and work efficiency.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0117] In this invention, the server includes means for analyzing video data, means for generating a communication method based on the analysis results, means for presenting the generated communication method to the terminal, means for creating and presenting feedback in real time based on the analysis results, and means for providing appropriate instructions according to the worker's situation. This makes it possible to accurately grasp the worker's emotions and psychological state and provide appropriate feedback and instructions based on that in real time.

[0118] "Video data" refers to video signals and image information acquired through devices such as cameras.

[0119] "Means for acquiring" refers to a method or device for acquiring video data using a device such as a camera.

[0120] "Server" refers to a central computing device for storing, processing, and analyzing data over a network.

[0121] "Means of analysis" refers to methods and software for extracting, evaluating, and determining specific information using acquired data.

[0122] "Means for generating communication methods" refers to systems or processes for creating appropriate dialogue methods or instructions based on the analysis results.

[0123] "Terminal" refers to a device or display device that can be directly operated by a user.

[0124] "Presenting means" refers to a method or device for visually or audibly presenting the generated information to the user.

[0125] "User" refers to a person or operator who uses the system.

[0126] "Feedback" refers to information such as evaluations, instructions, and advice provided to workers by the system.

[0127] "Appropriate instructions" refers to specific guidance and advice on the work content that is provided immediately based on the analysis results and in accordance with the situation and the worker's condition.

[0128] To implement this invention, a camera for acquiring video data, a server for processing and storing the data, and a terminal for providing feedback to the user must all work together. The specific operation of this system will now be described.

[0129] Device behavior

[0130] The terminal first starts operation when the user starts. A camera is connected to the terminal, and this camera is used to capture video data of the user and workers in real time. The captured video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[0131] Server Operation

[0132] The server receives the video data sent from the device and instantly analyzes it using a generative AI model. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, to infer the worker's emotions and psychological state. Specifically, the generative AI model identifies states such as happiness, tension, and concentration, and generates work instructions and feedback based on these. The analysis results are stored in a database and made available for immediate use.

[0133] Generating and Presenting Feedback

[0134] The server generates effective feedback based on the analysis results. This generated feedback includes multiple patterns, such as encouraging the worker to "take a slow, deep breath" if they are feeling nervous, or instructions to "take your time and work carefully." The generated feedback is sent to the device and displayed on the user's UI. The user can view this and select the most appropriate feedback. The user's choice from the displayed options is immediately applied, and the work continues.

[0135] Specific examples

[0136] For example, if a worker is operating a new machine on a factory production line, the device captures the worker's video data via a camera and sends it to a server. The server then analyzes the video data using a generative AI model and determines that the worker is nervous. Based on this, the server generates feedback such as "Take a deep breath and relax" or "Please work slowly." These options are presented to the user via the device, allowing them to choose the appropriate method to support the worker.

[0137] Prompt Sentence Examples

[0138] Here are some example prompts to input to a generative AI model:

[0139] Analyze the worker's facial expressions, gestures, complexion, and gaze from the input image to infer the worker's emotions and psychological state. Then, generate the following feedback: reassuring words if the worker is nervous, light-hearted conversation if the worker is relaxed, and encouraging words to keep working if the worker is concentrating.

[0140] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0141] Step 1:

[0142] The device uses a camera to capture video data in real time.

[0143] The input is the live video from the camera, and the output is the captured video data. Specifically, the camera sensor captures the light signal and converts it into digital video data.

[0144] Step 2:

[0145] The video data acquired by the terminal is sent to the server at regular intervals.

[0146] The input is the captured video data, and the output is the data sent to the server. Specifically, the operation is to use a network protocol to configure the video data into packets and send them to the server.

[0147] Step 3:

[0148] The video data received by the server is analyzed using a generative AI model.

[0149] The input is video data sent from the device, and the output is analyzed data on emotions and psychological states. Specifically, the system analyzes the video data frame by frame, identifies facial expressions, gestures, complexion, and gaze, and processes the data to determine emotions and feelings.

[0150] Step 4:

[0151] The server generates feedback based on the analysis results.

[0152] The input is the analyzed emotional and psychological data, and the output is the generated feedback. Specifically, the analysis results are used as metadata, and prompt sentences are input to the generative AI model to generate appropriate feedback.

[0153] Step 5:

[0154] The server transmits the generated feedback to the terminal.

[0155] The input is the generated feedback, and the output is the feedback sent to the terminal. The specific operation is to configure the feedback as a packet via a network protocol and send it to the terminal.

[0156] Step 6:

[0157] The device presents the received feedback on the user's UI.

[0158] The input is the feedback sent from the server, and the output is the feedback presented to the user. The specific operation is to process the received data and display it in an appropriate format on the user interface.

[0159] Step 7:

[0160] The user selects the feedback provided.

[0161] The input is multiple feedback options presented on the UI, and the output is the selected feedback. The specific operation is to accept the user's operation, record the selection result, and save it in a form that can be used for subsequent processing.

[0162] Step 8:

[0163] Provide the worker with user-selected feedback.

[0164] The input is the selected feedback, and the output is the feedback provided to the worker, which can be communicated to the worker via a terminal or smart glasses in the form of voice or text.

[0165] This series of processing steps enables the system to provide appropriate feedback in real time according to the worker's emotions and psychological state.

[0166] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0167] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0168] Device behavior

[0169] The device starts operating when a user starts a remote conference. A camera and microphone are connected to the device, and they are used to capture video data of the user and the other party in real time. The captured video data is encoded frame by frame and temporarily stored in a buffer. This video data is then sent to the server at regular intervals.

[0170] Server Operation

[0171] The server receives the video data sent from the device and immediately begins analysis. This analysis uses generative AI and an emotion engine. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze. The emotion engine uses this data to recognize the user's emotions in real time.

[0172] Emotion Engine

[0173] The emotion engine uses machine learning algorithms to accurately determine a user's emotions. It combines facial expression analysis, gesture analysis, facial color analysis, and eye gaze analysis to identify states such as happiness, tension, and concentration. The results are immediately stored on the server and used as analysis results.

[0174] Creation and presentation of communication methods

[0175] The server generates multiple effective communication methods based on the analysis results of the emotion engine, such as light-hearted conversation if the user is relaxed, or reassuring words if the user is nervous.

[0176] The generated communication methods are sent to the device in real time and displayed on the user's UI. The user can select the appropriate option from these options and proceed with the conversation accordingly. This selection enables smooth communication that is close to face-to-face.

[0177] Specific examples

[0178] For example, consider a case where a user is engaged in an online counseling session. In this case, the device captures video data of the counseling session via a camera and sends it to a server. The server analyzes the video data using generative AI and an emotion engine to determine that the client is nervous. Based on this determination, the server generates communication methods such as "I'll teach you breathing techniques to help you relax" or "You can speak more slowly." These options are presented to the user via the device, allowing them to choose the most appropriate method and smoothly progress with the counseling session.

[0179] Convenience

[0180] This system can be applied in a variety of situations, including in education, corporate interviews, and counseling. By providing real-time feedback based on analysis of non-verbal information and providing effective communication methods, it is possible to achieve high-quality dialogue similar to face-to-face, even in a remote environment.

[0181] In this way, the present invention can significantly improve the quality of remote communication and facilitate smooth dialogue.

[0182] The processing flow will be explained below.

[0183] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0184] Step 1:

[0185] A user starts a remote conference. Using a device (PC, smartphone, etc.), the user joins the conference URL on the ZOOM application or starts a new conference. At this time, make sure that the camera and microphone are connected correctly.

[0186] Step 2:

[0187] The device uses a camera to acquire video data in real time. The device uses the camera device to capture and encode video data for each frame. The encoded video data is temporarily stored in a buffer.

[0188] Step 3:

[0189] The video data acquired by the device is sent to a server at regular intervals. The video data is then uploaded to the server via the Internet. This process uses a dedicated API and data transmission protocol.

[0190] Step 4:

[0191] The server analyzes the received video data. The received data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. This analysis allows features to be extracted from the data.

[0192] Step 5:

[0193] The server uses an emotion engine to recognize the user's emotions in real time. The emotion engine uses a machine learning algorithm based on the analyzed features to determine emotions such as happiness, tension, and concentration. The results of the determination are stored internally on the server.

[0194] Step 6:

[0195] The server generates multiple communication methods based on the results of the emotion engine. For example, if the user is relaxed, it generates methods such as "continue the conversation in a relaxed manner," and if the user is nervous, it generates methods such as "say something comforting."

[0196] Step 7:

[0197] The server sends the generated communication methods to the device. The sent data includes multiple suggested patterns (light jokes, words of encouragement, etc.). These communication methods are sent in real time.

[0198] Step 8:

[0199] The device then presents the received communication method to the user, who then selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option.

[0200] Step 9:

[0201] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[0202] In this way, the system of the present invention can effectively acquire non-verbal information in remote conferences just as it does in face-to-face conferences, and can improve the quality of dialogue by providing appropriate means of communication.

[0203] Example 2

[0204] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] In remote meetings and interviews, it is difficult to accurately grasp the user's emotions and psychological state and communicate effectively. In addition, there is a lack of systems that can analyze non-verbal information and provide appropriate communication methods in real time.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0207] In this invention, the server includes means for acquiring visual and audio data from the user, means for transmitting the acquired visual and audio data to the server, means for the server to analyze the visual and audio data, means for generating appropriate communication means based on the analysis results, means for presenting the generated communication means to the terminal, and means for the user to select the presented communication means. This makes it possible to analyze the user's emotions and psychological state in real time and provide appropriate communication means.

[0208] "Visual data" is data in digital form that contains images of the user and their counterparts.

[0209] "Voice data" is audio data that includes the pronunciation and conversation content of the user and the other party.

[0210] "Means for acquiring" refers to devices and methods for acquiring data using a camera and microphone.

[0211] "Means for sending to the server" refers to the communication protocol and infrastructure for transferring the acquired data to the server.

[0212] "Means for analyzing" means software and hardware for processing and analyzing acquired visual and audio data.

[0213] "Means for generating appropriate communication methods" are algorithms or models that suggest effective conversations and actions based on the analysis results.

[0214] The "means for presenting on the terminal" is an interface for visually displaying the generated communication means to the user.

[0215] "User-selected means" refers to an interface that allows the user to select and execute an appropriate communication means from the presented means.

[0216] An "expression" is a visual expression of emotion expressed by the movement of specific muscles in the face.

[0217] "Movement" is behavior data that indicates the user's physical movements and posture.

[0218] "Skin tone" is data that represents changes in the complexion and skin color of the user.

[0219] "Gaze" is data that includes the user's eye movements and gaze points.

[0220] MODE FOR CARRYING OUT THE INVENTION

[0221] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0222] Device behavior

[0223] The device starts operating when a user starts a remote conference. The device is equipped with a camera and microphone, which are used to capture video data of the user and the other party in real time. The captured video data is processed frame by frame using encoding methods such as MJPEG or H.264 and temporarily stored in a buffer. This processing is often performed using OpenCV.

[0224] For example, when a user is using a web conferencing tool (e.g., an online meeting system), the camera and microphone are activated to capture video and audio. This video data is captured at 30 frames per second and stored in a buffer.

[0225] Sending data to the server

[0226] The device sends the encoded video data to the server at regular intervals using protocols such as WebSocket or RTMP (Real-Time Messaging Protocol), which allows data to be transferred with low latency.

[0227] Example: Video data captured on a device is sent to a server once per second. The protocol used is WebSocket, which enables real-time transmission.

[0228] Server-side data analysis

[0229] The server receives the video data sent from the device and immediately begins analysis. Server-side processing uses image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch.

[0230] Example: The server analyzes the frames of video data received using OpenCV to extract features such as gestures and facial expressions. These features are then further analyzed using a TensorFlow model to obtain more detailed information.

[0231] Emotion recognition by emotion engine

[0232] The server uses generative AI and an emotion engine to analyze the user's emotions from video data in real time. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze, and the emotion engine recognizes the user's emotional state based on that data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests.

[0233] Example: Facial expression analysis can quantify humorous reactions and levels of tension. For example, when a significant frown or lip curl is detected, that information is sent to an emotion engine, which then determines the emotional state, such as "tension" or "anxiety."

[0234] Creating communication methods

[0235] Based on the analysis results from the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods. Depending on the analysis results, it generates conversations that incorporate light-hearted jokes if the user is relaxed, or words that reassure the user if they are nervous.

[0236] Example: If a user is determined to be nervous, GPT-3 will generate advice such as "Take a deep breath and you'll feel better."

[0237] Example prompt sentence:

[0238] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[0239] Providing communication methods

[0240] The server sends the generated communication method to the device in real time and displays it on the user's UI via a REST API or notification service.

[0241] Example: The generated advice is displayed in a pop-up format in the upper right corner of the screen, making it easy for users to see.

[0242] User selection and execution

[0243] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that, enabling smooth communication that is close to face-to-face.

[0244] Example: A user selects the advice "Take a deep breath and feel better" and verbally repeats it to the other person, making the conversation flow more naturally and effectively.

[0245] This system analyzes emotions in real time and provides effective communication methods, enabling high-quality dialogue even in remote environments. This process makes remote meetings and interviews as smooth as face-to-face meetings.

[0246] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0247] Step 1:

[0248] When a user starts a remote conference, the device activates the camera and microphone and acquires visual and audio data in real time. The device uses OpenCV to capture video data frame by frame (e.g., 30 frames per second) and stores audio data in a buffer. The acquired visual and audio data are input.

[0249] Specific operation: The camera captures the user's face, the microphone collects the user's speech, and the data is temporarily stored in the device's memory.

[0250] Input: User's visual and audio data

[0251] Output: Buffered encoded visual and audio data

[0252] Step 2:

[0253] The device transmits the visual and audio data stored in the buffer to the server at regular intervals (e.g., once per second) using a low-latency protocol such as WebSocket or RTMP.

[0254] Specific operation: The device opens a WebSocket and sends video frames and audio samples as binary data to the server.

[0255] Input: Buffered encoded visual and audio data

[0256] Output: Visual and audio data sent to the server

[0257] Step 3:

[0258] The server analyzes the visual and audio data received from the device. Image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch are used for server-side processing. The received data is first decoded and then analyzed to extract visual and audio features.

[0259] Specific operation: The server decodes and decomposes the received data into frames, and uses OpenCV to extract features such as face and eye direction. At the same time, it also analyzes the features of the audio data.

[0260] Input: Visual and audio data sent to the server

[0261] Output: Analyzed visual and audio feature data

[0262] Step 4:

[0263] The server uses generative AI and an emotion engine to recognize the user's emotions in real time from the analyzed visual and audio feature data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests to determine emotional states such as tension, joy, and concentration.

[0264] Specific operation: The emotion engine uses a neural network to analyze visual data such as facial expressions, gestures, skin tone, and gaze, and then combines this data with a machine learning algorithm to determine emotions.

[0265] Input: Analyzed visual and audio feature data

[0266] Output: Determined user's emotional state

[0267] Step 5:

[0268] Based on the analysis results of the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods, such as light-hearted banter when the user is relaxed, or reassuring words when the user is nervous.

[0269] Specific operation: A generative AI model such as GPT-3 generates appropriate dialogue content using prompt sentences based on the results of sentiment analysis.

[0270] Example prompt sentence:

[0271] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[0272] Input: Determined emotional state of the user

[0273] Output: The generated means of communication

[0274] Step 6:

[0275] The server sends the generated communication methods to the device in real time and displays them on the user's UI. This transmission uses a REST API or notification service.

[0276] Specific operation: The server converts the generated communication method into JSON format, sends it to the terminal via an HTTP request, and presents it on the terminal in a pop-up or dialog format.

[0277] Input: Generated communication method

[0278] Output: Communication options displayed on the device

[0279] Step 7:

[0280] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that choice, achieving smooth communication that is close to face-to-face.

[0281] Specific behavior: The user clicks or taps on an option presented in the UI and continues the conversation with the other person using that method.

[0282] Input: Communication options displayed on the device

[0283] Output: Dialogue content based on the communication method selected by the user

[0284] (Application example 2)

[0285] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] In traditional remote classes, it was difficult for teachers to instantly grasp students' emotional states and provide effective feedback at the appropriate time. Furthermore, the inability to provide appropriate explanations tailored to students' situations could result in a decline in educational effectiveness. This resulted in the quality of remote classes being inferior to face-to-face classes.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0288] In this invention, the server includes: [means for transmitting acquired video data to the server;] [means for the server to analyze the video data;] [means for generating a communication method based on the analysis results;] [means for using smart glasses in an educational setting to present the generated communication method to a teacher in real time as feedback according to the student's emotional state; and [means for generating prompt sentences according to the emotional state and providing effective feedback.] This enables teachers to grasp students' emotional states in real time and provide appropriate feedback and supplementary explanations.

[0289] "Video data" refers to information in video frames captured using imaging equipment such as a camera.

[0290] A "server" is a computer system that processes and analyzes data over a network.

[0291] "Analysis" is the act of breaking down the information from the acquired data and examining its contents in detail.

[0292] "Generation" is the act of creating new results or means based on data or information.

[0293] A "terminal" is a hardware device that is directly operated by a user.

[0294] A "communication method" is a means or approach for effective dialogue or communication.

[0295] "Smart glasses" are glasses-type devices equipped with a camera and a display that can provide visual information.

[0296] "Feedback" is the act of returning information for the purpose of improvement or correction based on the information or results obtained.

[0297] A "prompt sentence" is an input sentence that is used to provide guidance and help to the generative AI model.

[0298] The "educational setting" refers to the place or environment where lessons and learning activities take place.

[0299] A "user" is a person or entity that uses a system or device.

[0300] "Emotional state" refers to the user's psychological state and fluctuations in feelings.

[0301] A system for specifically implementing this invention uses smart glasses, a camera, a server, a generative AI model, and an emotion engine, and the following embodiment is described in detail below.

[0302] First, the smart glasses are equipped with a camera that captures students' facial expressions, gestures, and gaze in real time during remote classes. The captured video data is temporarily stored in a buffer inside the smart glasses and then periodically sent to a server.

[0303] The server analyzes the received video data. The analysis includes a generative AI model and an emotion engine that recognizes the student's emotional state in real time based on facial expressions, gestures, complexion, gaze, and other factors. The generative AI model and emotion engine can accurately determine whether the student is in an emotional state such as happiness, confusion, or concentration.

[0304] Next, based on the analysis results, the server generates multiple effective communication methods. These methods are presented in real time to the teacher wearing smart glasses. For example, if a student is confused, the teacher will provide feedback such as "Please explain again using easy-to-understand examples and metaphors."

[0305] Teachers can refer to these provided feedback and communication methods as they conduct their lessons, enabling them to provide feedback at the appropriate time based on students' level of understanding and emotional state, thereby improving the effectiveness of remote lessons.

[0306] Hardware and software used

[0307] Hardware: Smart glasses (with camera)

[0308] Software: OpenCV, EmotionRecognition library, CommunicationSuggestions module

[0309] Specific examples

[0310] For example, if the emotion engine recognizes that a student is confused during a lesson, the server will provide the teacher with feedback such as, "If the student is confused -> Explain again using easy-to-understand examples and metaphors." The teacher can then refer to this feedback and rephrase the problem using a concrete example, such as, "This math problem might be easier to understand if you explain it using an apple."

[0311] Prompt Sentence Examples

[0312] "If students are confused, explain it again with an alternative, clearer example."

[0313] "If students are focused, keep them on pace."

[0314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0315] Step 1:

[0316] The camera in the smart glasses captures video data of the students.

[0317] (Input) Real-time video of the student.

[0318] (Data processing) The camera sensor captures video frames and temporarily stores them in a buffer.

[0319] (Output) Video data for each frame.

[0320] Step 2:

[0321] The video data acquired by the smart glasses is sent to the server.

[0322] (Input) Video data stored in the buffer.

[0323] (Data processing) Video data is sent to the server reliably using network protocols.

[0324] (Output) Video data received by the server.

[0325] Step 3:

[0326] The server analyzes the received video data using a generative AI model and emotion engine.

[0327] (Input) Video data received by the server.

[0328] (Data processing) The generative AI model analyzes facial expressions, gestures, complexion, gaze, etc. for each frame, and the emotion engine uses this information to infer the student's emotional state.

[0329] (Output) Inferred emotional state (e.g., happy, confused, focused, etc.).

[0330] Step 4:

[0331] The server generates a plurality of effective communication methods based on the inferred emotional state.

[0332] (Input) Inferred emotional state.

[0333] (Data processing) Generate prompt sentences that correspond to the emotional state, and generate effective feedback methods using a generative AI model.

[0334] (Output) Multiple communication methods generated.

[0335] Step 5:

[0336] The server presents the generated communication method to the smart glasses in real time.

[0337] (Input) Multiple communication methods generated.

[0338] (Data processing) Displayed as text data on the smart glasses display.

[0339] (Output) Communication methods displayed on the smart glasses display.

[0340] Step 6:

[0341] The user, the teacher, selects the communication method presented and applies it to their lessons.

[0342] (Input) Communication methods displayed on the smart glasses display.

[0343] (Data processing) Teachers can choose to provide feedback at appropriate times according to students' emotional state.

[0344] (Output) Appropriate feedback from the teacher to the student.

[0345] Examples:

[0346] Example prompt sentence:

[0347] "If students are confused, explain it again with an alternative, clearer example."

[0348] "If students are focused, keep them on pace."

[0349] In this way, the system operates while clearly defining the specific input, data processing, and output at each step, allowing teachers to provide appropriate feedback to students in real time during remote classes based on their emotional state.

[0350] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0351] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0352] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0353] [Second embodiment]

[0354] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0355] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0356] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0357] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0358] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0359] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0360] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0361] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0362] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0363] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0364] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0365] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0366] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[0367] Device behavior

[0368] The terminal first starts operating when a user starts a remote conference. A camera is connected to the terminal, and video data of the user and the other party is acquired in real time using this camera. The acquired video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[0369] Server Operation

[0370] The server receives the video data sent from the device and instantly analyzes it using generative AI. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, and uses these to infer the other person's emotions and psychological state. Specifically, the generative AI identifies states such as happiness, tension, and concentration, and generates a communication method based on these. The analysis results are stored in a database and made available for immediate use.

[0371] Creation and presentation of communication methods

[0372] The server generates several effective communication methods based on the analysis results. The generated communication methods include multiple patterns, such as light-hearted conversations when the other person is relaxed, or reassuring words when the other person is nervous.

[0373] The generated communication methods are sent to the terminal and presented on the user's UI. The user can view the options and select the most suitable communication method. The user's choice from the displayed options is immediately applied, and the conversation continues.

[0374] Specific examples

[0375] For example, consider a case where a user is conducting a remote job interview. In this case, the device captures video data of the interviewer via a camera and sends it to a server. The server uses generative AI to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier." These options are presented to the user via the device, allowing them to choose the appropriate method and smoothly progress through the interview.

[0376] In this way, the present invention makes it possible to achieve effective communication in remote conferences that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[0377] The processing flow will be explained below.

[0378] Step 1:

[0379] A user starts a remote conference. The user uses a device (such as a PC or smartphone) to join the ZOOM conference using a camera and microphone. The user joins the conference URL on the ZOOM application or starts a new conference.

[0380] Step 2:

[0381] The device captures video data in real time using a camera, encodes the captured video data frame by frame, and temporarily stores it in a buffer. This video data is then sent to a server at regular intervals.

[0382] Step 3:

[0383] The server analyzes the received video data. The video data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. As a result of the analysis, the other person's emotions and psychological state are inferred.

[0384] Step 4:

[0385] The server generates several communication methods based on the analysis results. For example, if the person smiles a lot, it will judge them to be "relaxed," and if they have furrowed brows, it will judge them to be "tense," and generate a communication method appropriate to each.

[0386] Step 5:

[0387] The server generates a communication method and sends it to the device in real time, including multiple suggested patterns (e.g., humorous conversation, words of encouragement, etc.).

[0388] Step 6:

[0389] The device then presents the received communication method to the user. The user selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option. This selection allows for smooth communication that is close to face-to-face.

[0390] Step 7:

[0391] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[0392] In this way, the system of the present invention can effectively obtain non-verbal information in remote conferences similar to that in face-to-face conferences and provide appropriate means of communication, thereby improving the quality of dialogue.

[0393] Example 1

[0394] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0395] In remote conferences, non-face-to-face communication between participants poses the challenge of accurately predicting the other person's emotions and psychological state from their facial expressions and gestures, and using appropriate communication methods. As a result, the effectiveness of communication decreases, and the possibility of misunderstandings and breakdowns in dialogue increases.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0397] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to a data processing device, and means for the data processing device to analyze the video data. This enables the user to analyze the emotions and psychological state of the other person and generate and present an effective communication method based on that analysis.

[0398] "Video data" refers to visual information captured using a camera or other optical device.

[0399] A "data processing device" is a device for analyzing video data and processing the information using specific algorithms.

[0400] "Analyzing" is the process of using captured video data to detect and evaluate specific patterns or characteristics.

[0401] "Communication methods" refer to the means and approaches of dialogue that are generated based on the emotions and psychological state of the other person.

[0402] A "display device" is a device for visually presenting the generated communication method to the user.

[0403] "User" refers to an individual or group that operates this system and interacts with it through a terminal.

[0404] A "means" is a device, process, or system element designed to accomplish a particular purpose.

[0405] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[0406] Device behavior

[0407] The device first begins operation when a user starts a remote conference. The specific hardware used is a camera (e.g., a standard webcam). This camera is used to capture video data of the user and the other party in real time. The captured video data is captured at a rate of 30 frames per second and compressed and encoded using FFmpeg software. The encoded video data is then sent to the server using WebSocket.

[0408] Server Operation

[0409] The server receives the video data sent from the device and immediately begins analysis. For analysis, a generative AI model (e.g., a general generative AI model and TensorFlow) is used. This generative AI extracts features such as facial expressions, gestures, complexion, and gaze from the video data, and infers psychological states such as happiness, tension, and concentration. The analysis results are stored in a database (e.g., a general relational database or NoSQL database).

[0410] Creation and presentation of communication methods

[0411] The server generates several effective communication methods based on the analysis results. This generation uses a generative AI model (e.g., a general generative AI model) with natural language generation (NLG) capabilities. The generated communication methods include multiple patterns, such as light-hearted banter when the other person is relaxed, or reassuring words when the other person is nervous. Below is an example of a prompt:

[0412] Sample prompt: "Generate appropriate communication suggestions if the other person is nervous."

[0413] The generated communication method is converted into JSON format and sent to the terminal via WebSocket communication.

[0414] Display on the device's user interface

[0415] The device presents the received communication method on a user interface (UI), for example, displaying options such as "Recommend deep breathing to relax" or "Start with a simple question to make it easier to talk."

[0416] User selection and application

[0417] The user selects the communication method that they think is most appropriate from the options displayed on the device's UI. The selected method is immediately applied and the conversation continues.

[0418] Specific examples

[0419] For example, when a user conducts a remote job interview, the device uses a camera to capture video data of the interviewer and sends it to a server. The server uses a generative AI model to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier to understand," and sends these to the device. The device then presents these options on a user interface, allowing the user to select the appropriate method to smoothly proceed with the interview.

[0420] Through this procedure, the present invention enables effective communication in remote meetings that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[0421] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0422] Step 1:

[0423] A user starts the meeting software on a terminal and starts a remote conference. The input is the user's operation (a conference start action), and the output is the system entering an operational state.

[0424] Step 2:

[0425] The device uses a camera to capture real-time video data of the user and the target. The input is the video signal from the camera, and the output is the captured video data. FFmpeg is used to compress and encode the video data.

[0426] Step 3:

[0427] The device sends the encoded video data to the server using WebSocket communication. The input is the encoded video data, and the output is data packets via WebSocket.

[0428] Step 4:

[0429] The server receives video data sent from the device. The input is data packets sent over WebSocket, and the output is parseable video data, which the server stores in a buffer.

[0430] Step 5:

[0431] The server analyzes the video data using a generative AI model. The input is the received video data, and the output is characteristic data such as facial expressions, gestures, complexion, and gaze. Based on this data, the server processes the other person's emotions and psychological state.

[0432] Step 6:

[0433] The server generates several effective communication methods based on the analysis results. The input is the analysis results (emotions and psychological state), and the output is the generated multiple communication methods. A generative AI model with natural language generation (NLG) capabilities is used for generation.

[0434] Step 7:

[0435] The server converts the generated communication methods into JSON format and sends it to the terminal via WebSocket communication. The input is multiple communication methods (text data), and the output is JSON data, which is sent to the terminal via WebSocket.

[0436] Step 8:

[0437] The device parses the received JSON data and displays the generated communication methods on the user interface (UI). The input is JSON-formatted data, and the output is options on the UI. For example, options such as "Recommend deep breathing" or "Ask a simple question" are displayed.

[0438] Step 9:

[0439] The user selects the most appropriate communication method from the displayed options. The input is the UI option, and the output is the selected communication method. The selected method is applied immediately.

[0440] Step 10:

[0441] The device reports the selected communication method to the server and the dialogue continues. The input is the communication method selected by the user, and the output is the selection sent to the server, which uses this to prepare for the next analysis cycle.

[0442] (Application example 1)

[0443] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0444] In industrial settings, there is a need to grasp the emotions and psychological state of workers in real time and provide appropriate feedback accordingly. However, current systems have difficulty in doing so effectively, which can lead to a decline in worker efficiency and performance. In addition, there is a lack of means to provide instructions tailored to specific situations in real time, resulting in a lack of immediate response capabilities on-site. This poses challenges for improving worker safety and work efficiency.

[0445] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0446] In this invention, the server includes means for analyzing video data, means for generating a communication method based on the analysis results, means for presenting the generated communication method to the terminal, means for creating and presenting feedback in real time based on the analysis results, and means for providing appropriate instructions according to the worker's situation. This makes it possible to accurately grasp the worker's emotions and psychological state and provide appropriate feedback and instructions based on that in real time.

[0447] "Video data" refers to video signals and image information acquired through devices such as cameras.

[0448] "Means for acquiring" refers to a method or device for acquiring video data using a device such as a camera.

[0449] "Server" refers to a central computing device for storing, processing, and analyzing data over a network.

[0450] "Means of analysis" refers to methods and software for extracting, evaluating, and determining specific information using acquired data.

[0451] "Means for generating communication methods" refers to systems or processes for creating appropriate dialogue methods or instructions based on the analysis results.

[0452] "Terminal" refers to a device or display device that can be directly operated by a user.

[0453] "Presenting means" refers to a method or device for visually or audibly presenting the generated information to the user.

[0454] "User" refers to a person or operator who uses the system.

[0455] "Feedback" refers to information such as evaluations, instructions, and advice provided to workers by the system.

[0456] "Appropriate instructions" refers to specific guidance and advice on the work content that is provided immediately based on the analysis results and in accordance with the situation at hand and the worker's condition.

[0457] To implement this invention, a camera for acquiring video data, a server for processing and storing the data, and a terminal for providing feedback to the user must all work together. The specific operation of this system will now be described.

[0458] Device behavior

[0459] The terminal first starts operation when the user starts. A camera is connected to the terminal, and this camera is used to capture video data of the user and workers in real time. The captured video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[0460] Server Operation

[0461] The server receives the video data sent from the device and instantly analyzes it using a generative AI model. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, to infer the worker's emotions and psychological state. Specifically, the generative AI model identifies states such as happiness, tension, and concentration, and generates work instructions and feedback based on these. The analysis results are stored in a database and made available for immediate use.

[0462] Generating and Presenting Feedback

[0463] The server generates effective feedback based on the analysis results. This generated feedback includes multiple patterns, such as encouraging the worker to "take a slow, deep breath" if they are feeling nervous, or instructions to "take your time and work carefully." The generated feedback is sent to the device and displayed on the user's UI. The user can view this and select the most appropriate feedback. The user's choice from the displayed options is immediately applied, and the work continues.

[0464] Specific examples

[0465] For example, if a worker is operating a new machine on a factory production line, the device captures the worker's video data via a camera and sends it to a server. The server then analyzes the video data using a generative AI model and determines that the worker is nervous. Based on this, the server generates feedback such as "Take a deep breath and relax" or "Please work slowly." These options are presented to the user via the device, allowing them to choose the appropriate method to support the worker.

[0466] Prompt Sentence Examples

[0467] Here are some example prompts to input to a generative AI model:

[0468] Analyze the worker's facial expressions, gestures, complexion, and gaze from the input image to infer the worker's emotions and psychological state. Then, generate the following feedback: reassuring words if the worker is nervous, light-hearted conversation if the worker is relaxed, and encouraging words to keep working if the worker is concentrating.

[0469] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0470] Step 1:

[0471] The device uses a camera to capture video data in real time.

[0472] The input is the live video from the camera, and the output is the captured video data. Specifically, the camera sensor captures the light signal and converts it into digital video data.

[0473] Step 2:

[0474] The video data acquired by the terminal is sent to the server at regular intervals.

[0475] The input is the captured video data, and the output is the data sent to the server. Specifically, the operation is to use a network protocol to configure the video data into packets and send them to the server.

[0476] Step 3:

[0477] The video data received by the server is analyzed using a generative AI model.

[0478] The input is video data sent from the device, and the output is analyzed data on emotions and psychological states. Specifically, the system analyzes the video data frame by frame, identifies facial expressions, gestures, complexion, and gaze, and processes the data to determine emotions and feelings.

[0479] Step 4:

[0480] The server generates feedback based on the analysis results.

[0481] The input is the analyzed emotional and psychological data, and the output is the generated feedback. Specifically, the analysis results are used as metadata, and prompt sentences are input to the generative AI model to generate appropriate feedback.

[0482] Step 5:

[0483] The server transmits the generated feedback to the terminal.

[0484] The input is the generated feedback, and the output is the feedback sent to the terminal. The specific operation is to configure the feedback as a packet via a network protocol and send it to the terminal.

[0485] Step 6:

[0486] The device presents the received feedback on the user's UI.

[0487] The input is the feedback sent from the server, and the output is the feedback presented to the user. The specific operation is to process the received data and display it in an appropriate format on the user interface.

[0488] Step 7:

[0489] The user selects the feedback provided.

[0490] The input is multiple feedback options presented on the UI, and the output is the selected feedback. The specific operation is to accept the user's operation, record the selection result, and save it in a form that can be used for subsequent processing.

[0491] Step 8:

[0492] Provide the worker with user-selected feedback.

[0493] The input is the selected feedback, and the output is the feedback provided to the worker, which can be communicated to the worker via a terminal or smart glasses in the form of voice or text.

[0494] This series of processing steps enables the system to provide appropriate feedback in real time according to the worker's emotions and psychological state.

[0495] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0496] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0497] Device behavior

[0498] The device starts operating when a user starts a remote conference. A camera and microphone are connected to the device, and they are used to capture video data of the user and the other party in real time. The captured video data is encoded frame by frame and temporarily stored in a buffer. This video data is then sent to the server at regular intervals.

[0499] Server Operation

[0500] The server receives the video data sent from the device and immediately begins analysis. This analysis uses generative AI and an emotion engine. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze. The emotion engine uses this data to recognize the user's emotions in real time.

[0501] Emotion Engine

[0502] The emotion engine uses machine learning algorithms to accurately determine a user's emotions. It combines facial expression analysis, gesture analysis, facial color analysis, and eye gaze analysis to identify states such as happiness, tension, and concentration. The results are immediately stored on the server and used as analysis results.

[0503] Creation and presentation of communication methods

[0504] The server generates multiple effective communication methods based on the analysis results of the emotion engine, such as light-hearted conversation if the user is relaxed, or reassuring words if the user is nervous.

[0505] The generated communication methods are sent to the device in real time and displayed on the user's UI. The user can select the appropriate option from these options and proceed with the conversation accordingly. This selection enables smooth communication that is close to face-to-face.

[0506] Specific examples

[0507] For example, consider a case where a user is engaged in an online counseling session. In this case, the device captures video data of the counseling session via a camera and sends it to a server. The server analyzes the video data using generative AI and an emotion engine to determine that the client is nervous. Based on this determination, the server generates communication methods such as "I'll teach you breathing techniques to help you relax" or "You can speak more slowly." These options are presented to the user via the device, allowing them to choose the most appropriate method and smoothly progress with the counseling session.

[0508] Convenience

[0509] This system can be applied in a variety of situations, including in education, corporate interviews, and counseling. By providing real-time feedback based on analysis of non-verbal information and providing effective communication methods, it is possible to achieve high-quality dialogue similar to face-to-face, even in a remote environment.

[0510] In this way, the present invention can significantly improve the quality of remote communication and facilitate smooth dialogue.

[0511] The processing flow will be explained below.

[0512] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0513] Step 1:

[0514] A user starts a remote conference. Using a device (PC, smartphone, etc.), the user joins the conference URL on the ZOOM application or starts a new conference. At this time, make sure that the camera and microphone are connected correctly.

[0515] Step 2:

[0516] The device uses a camera to acquire video data in real time. The device uses the camera device to capture and encode video data for each frame. The encoded video data is temporarily stored in a buffer.

[0517] Step 3:

[0518] The video data acquired by the device is sent to a server at regular intervals. The video data is then uploaded to the server via the Internet. A dedicated API and data transmission protocol are used for this purpose.

[0519] Step 4:

[0520] The server analyzes the received video data. The received data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. This analysis allows features to be extracted from the data.

[0521] Step 5:

[0522] The server uses an emotion engine to recognize the user's emotions in real time. The emotion engine uses a machine learning algorithm based on the analyzed features to determine emotions such as happiness, tension, and concentration. The results of the determination are stored internally on the server.

[0523] Step 6:

[0524] The server generates multiple communication methods based on the results of the emotion engine. For example, if the user is relaxed, it generates methods such as "continue the conversation in a relaxed manner," and if the user is nervous, it generates methods such as "say something comforting."

[0525] Step 7:

[0526] The server sends the generated communication methods to the device. The sent data includes multiple suggested patterns (light jokes, words of encouragement, etc.). These communication methods are sent in real time.

[0527] Step 8:

[0528] The device then presents the received communication method to the user, who then selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option.

[0529] Step 9:

[0530] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[0531] In this way, the system of the present invention can effectively acquire non-verbal information in remote conferences just as it does in face-to-face conferences, and can improve the quality of dialogue by providing appropriate means of communication.

[0532] Example 2

[0533] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0534] In remote meetings and interviews, it is difficult to accurately grasp the user's emotions and psychological state and communicate effectively. In addition, there is a lack of systems that can analyze non-verbal information and provide appropriate communication methods in real time.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0536] In this invention, the server includes means for acquiring visual and audio data from the user, means for transmitting the acquired visual and audio data to the server, means for the server to analyze the visual and audio data, means for generating appropriate communication means based on the analysis results, means for presenting the generated communication means to the terminal, and means for the user to select the presented communication means. This makes it possible to analyze the user's emotions and psychological state in real time and provide appropriate communication means.

[0537] "Visual data" is data in digital form that contains images of the user and their counterparts.

[0538] "Voice data" is audio data that includes the pronunciation and conversation content of the user and the other party.

[0539] "Means for acquiring" refers to devices and methods for acquiring data using a camera and microphone.

[0540] "Means for sending to the server" refers to the communication protocol and infrastructure for transferring the acquired data to the server.

[0541] "Means for analyzing" means software and hardware for processing and analyzing acquired visual and audio data.

[0542] "Means for generating appropriate communication methods" are algorithms or models that suggest effective conversations and actions based on the analysis results.

[0543] The "means for presenting on the terminal" is an interface for visually displaying the generated communication means to the user.

[0544] "User-selected means" refers to an interface that allows the user to select and execute an appropriate communication means from the presented means.

[0545] An "expression" is a visual expression of emotion expressed by the movement of specific muscles in the face.

[0546] "Movement" is behavior data that indicates the user's physical movements and posture.

[0547] "Skin tone" is data that represents changes in the complexion and skin color of the user.

[0548] "Gaze" is data that includes the user's eye movements and gaze points.

[0549] MODE FOR CARRYING OUT THE INVENTION

[0550] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0551] Device behavior

[0552] The device starts operating when a user starts a remote conference. The device is equipped with a camera and microphone, which are used to capture video data of the user and the other party in real time. The captured video data is processed frame by frame using encoding methods such as MJPEG or H.264 and temporarily stored in a buffer. This processing is often performed using OpenCV.

[0553] Example: When a user is using a web conferencing tool (e.g., an online meeting system), the camera and microphone are activated to capture video and audio. This video data is captured at 30 frames per second and stored in a buffer.

[0554] Sending data to the server

[0555] The device sends the encoded video data to the server at regular intervals using protocols such as WebSocket or RTMP (Real-Time Messaging Protocol), which allows data to be transferred with low latency.

[0556] Example: Video data captured on a device is sent to a server once per second. The protocol used is WebSocket, which enables real-time transmission.

[0557] Server-side data analysis

[0558] The server receives the video data sent from the device and immediately begins analysis. Server-side processing uses image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch.

[0559] Example: The server analyzes the frames of video data received using OpenCV to extract features such as gestures and facial expressions. These features are then further analyzed using a TensorFlow model to obtain more detailed information.

[0560] Emotion recognition by emotion engine

[0561] The server uses generative AI and an emotion engine to analyze the user's emotions from video data in real time. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze, and the emotion engine recognizes the user's emotional state based on that data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests.

[0562] Example: Facial expression analysis can quantify humorous reactions and levels of tension. For example, if a significant frown or lip twist is detected, that information is sent to an emotion engine, which then determines the emotional state, such as "tension" or "anxiety."

[0563] Creating communication methods

[0564] Based on the analysis results from the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods. Depending on the analysis results, it generates conversations that incorporate light-hearted jokes if the user is relaxed, or words that reassure the user if they are nervous.

[0565] Example: If a user is determined to be nervous, GPT-3 will generate advice such as "Take a deep breath and you'll feel better."

[0566] Example prompt sentence:

[0567] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[0568] Providing communication methods

[0569] The server sends the generated communication method to the device in real time and displays it on the user's UI via a REST API or notification service.

[0570] Example: The generated advice is displayed in a pop-up format in the upper right corner of the screen, making it easy for users to see.

[0571] User selection and execution

[0572] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that, enabling smooth communication that is close to face-to-face.

[0573] Example: A user selects the advice "Take a deep breath and you'll feel better" and verbally repeats it to the other person, making the conversation flow more naturally and effectively.

[0574] This system analyzes emotions in real time and provides effective communication methods, enabling high-quality dialogue even in remote environments. This process makes remote meetings and interviews as smooth as face-to-face meetings.

[0575] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0576] Step 1:

[0577] When a user starts a remote conference, the device activates the camera and microphone and acquires visual and audio data in real time. The device uses OpenCV to capture video data frame by frame (e.g., 30 frames per second) and stores audio data in a buffer. The acquired visual and audio data are input.

[0578] Specific operation: The camera captures the user's face, the microphone collects the user's speech, and the data is temporarily stored in the device's memory.

[0579] Input: User's visual and audio data

[0580] Output: Buffered encoded visual and audio data

[0581] Step 2:

[0582] The device transmits the visual and audio data stored in the buffer to the server at regular intervals (e.g., once per second) using a low-latency protocol such as WebSocket or RTMP.

[0583] Specific operation: The device opens a WebSocket and sends video frames and audio samples as binary data to the server.

[0584] Input: Buffered encoded visual and audio data

[0585] Output: Visual and audio data sent to the server

[0586] Step 3:

[0587] The server analyzes the visual and audio data received from the device. Image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch are used for server-side processing. The received data is first decoded and then analyzed to extract visual and audio features.

[0588] Specific operation: The server decodes and decomposes the received data into frames, and uses OpenCV to extract features such as face and eye direction. At the same time, it also analyzes the features of the audio data.

[0589] Input: Visual and audio data sent to the server

[0590] Output: Analyzed visual and audio feature data

[0591] Step 4:

[0592] The server uses generative AI and an emotion engine to recognize the user's emotions in real time from the analyzed visual and audio feature data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests to determine emotional states such as tension, joy, and concentration.

[0593] Specific operation: The emotion engine uses a neural network to analyze visual data such as facial expressions, gestures, skin tone, and gaze, and then combines this data with a machine learning algorithm to determine emotions.

[0594] Input: Analyzed visual and audio feature data

[0595] Output: Determined user's emotional state

[0596] Step 5:

[0597] Based on the analysis results of the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods, such as light-hearted banter when the user is relaxed, or reassuring words when the user is nervous.

[0598] Specific operation: A generative AI model such as GPT-3 generates appropriate dialogue content using prompt sentences based on the results of sentiment analysis.

[0599] Example prompt sentence:

[0600] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[0601] Input: Determined emotional state of the user

[0602] Output: The generated means of communication

[0603] Step 6:

[0604] The server sends the generated communication methods to the device in real time and displays them on the user's UI. This transmission uses a REST API or notification service.

[0605] Specific operation: The server converts the generated communication method into JSON format, sends it to the terminal via an HTTP request, and presents it on the terminal in a pop-up or dialog format.

[0606] Input: Generated communication method

[0607] Output: Communication options displayed on the device

[0608] Step 7:

[0609] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that choice, achieving smooth communication that is close to face-to-face.

[0610] Specific behavior: The user clicks or taps on an option presented in the UI and continues the conversation with the other person using that method.

[0611] Input: Communication options displayed on the device

[0612] Output: Dialogue content based on the communication method selected by the user

[0613] (Application example 2)

[0614] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0615] In traditional remote classes, it was difficult for teachers to instantly grasp students' emotional states and provide effective feedback at the appropriate time. Furthermore, the inability to provide appropriate explanations tailored to students' situations could result in a decline in educational effectiveness. This resulted in the quality of remote classes being inferior to face-to-face classes.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0617] In this invention, the server includes: [means for transmitting acquired video data to the server;] [means for the server to analyze the video data;] [means for generating a communication method based on the analysis results;] [means for using smart glasses in an educational setting to present the generated communication method to a teacher in real time as feedback according to the student's emotional state; and [means for generating prompt sentences according to the emotional state and providing effective feedback.] This enables teachers to grasp students' emotional states in real time and provide appropriate feedback and supplementary explanations.

[0618] "Video data" refers to information in video frames captured using imaging equipment such as a camera.

[0619] A "server" is a computer system that processes and analyzes data over a network.

[0620] "Analysis" is the act of breaking down the information from the acquired data and examining its contents in detail.

[0621] "Generation" is the act of creating new results or means based on data or information.

[0622] A "terminal" is a hardware device that is directly operated by a user.

[0623] A "communication method" is a means or approach for effective dialogue or communication.

[0624] "Smart glasses" are glasses-type devices equipped with a camera and a display that can provide visual information.

[0625] "Feedback" is the act of returning information for the purpose of improvement or correction based on the information or results obtained.

[0626] A "prompt sentence" is an input sentence that is used to provide guidance and help to the generative AI model.

[0627] The "educational setting" refers to the place or environment where lessons and learning activities take place.

[0628] A "user" is a person or entity that uses a system or device.

[0629] "Emotional state" refers to the user's psychological state and fluctuations in feelings.

[0630] A system for specifically implementing this invention uses smart glasses, a camera, a server, a generative AI model, and an emotion engine, and the following embodiment is described in detail below.

[0631] First, the smart glasses are equipped with a camera that captures students' facial expressions, gestures, and gaze in real time during remote classes. The captured video data is temporarily stored in a buffer inside the smart glasses and then periodically sent to a server.

[0632] The server analyzes the received video data. The analysis includes a generative AI model and an emotion engine that recognizes the student's emotional state in real time based on facial expressions, gestures, complexion, gaze, and other factors. The generative AI model and emotion engine can accurately determine whether the student is in an emotional state such as happiness, confusion, or concentration.

[0633] Next, based on the analysis results, the server generates multiple effective communication methods. These methods are presented in real time to the teacher wearing the smart glasses. For example, if a student is confused, the teacher will provide feedback such as "Please explain again using easy-to-understand examples and metaphors."

[0634] Teachers can refer to these provided feedback and communication methods as they conduct their lessons, enabling them to provide feedback at the appropriate time based on students' level of understanding and emotional state, thereby improving the effectiveness of remote lessons.

[0635] Hardware and software used

[0636] Hardware: Smart glasses (with camera)

[0637] Software: OpenCV, EmotionRecognition library, CommunicationSuggestions module

[0638] Specific examples

[0639] For example, if the emotion engine recognizes that a student is confused during a lesson, the server will provide the teacher with feedback such as, "If the student is confused, please explain again using easy-to-understand examples and metaphors." The teacher can then use this feedback to re-explain the problem using concrete examples, such as, "This math problem might be easier to understand if you explain it using an apple."

[0640] Prompt Sentence Examples

[0641] "If students are confused, explain it again with an alternative, clearer example."

[0642] "If students are focused, keep them on pace."

[0643] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0644] Step 1:

[0645] The camera in the smart glasses captures video data of the students.

[0646] (Input) Real-time video of the student.

[0647] (Data processing) The camera sensor captures video frames and temporarily stores them in a buffer.

[0648] (Output) Video data for each frame.

[0649] Step 2:

[0650] The video data acquired by the smart glasses is sent to the server.

[0651] (Input) Video data stored in the buffer.

[0652] (Data processing) Video data is sent to the server reliably using network protocols.

[0653] (Output) Video data received by the server.

[0654] Step 3:

[0655] The server analyzes the received video data using a generative AI model and emotion engine.

[0656] (Input) Video data received by the server.

[0657] (Data processing) The generative AI model analyzes facial expressions, gestures, complexion, gaze, etc. for each frame, and the emotion engine uses this information to infer the student's emotional state.

[0658] (Output) Inferred emotional state (e.g., happy, confused, focused, etc.).

[0659] Step 4:

[0660] The server generates a plurality of effective communication methods based on the inferred emotional state.

[0661] (Input) Inferred emotional state.

[0662] (Data processing) Generate prompt sentences that correspond to the emotional state, and generate effective feedback methods using a generative AI model.

[0663] (Output) Multiple communication methods generated.

[0664] Step 5:

[0665] The server presents the generated communication method to the smart glasses in real time.

[0666] (Input) Multiple communication methods generated.

[0667] (Data processing) Displayed as text data on the smart glasses display.

[0668] (Output) Communication methods displayed on the smart glasses display.

[0669] Step 6:

[0670] The user, the teacher, selects the communication method presented and applies it to their lessons.

[0671] (Input) Communication methods displayed on the smart glasses display.

[0672] (Data processing) Teachers can choose to provide feedback at appropriate times according to students' emotional state.

[0673] (Output) Appropriate feedback from the teacher to the student.

[0674] Examples:

[0675] Example prompt sentence:

[0676] "If students are confused, explain it again with an alternative, clearer example."

[0677] "If students are focused, keep them on pace."

[0678] In this way, the system operates while clearly defining the specific input, data processing, and output at each step, allowing teachers to provide appropriate feedback to students in real time during remote classes based on their emotional state.

[0679] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0680] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0681] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0682] [Third embodiment]

[0683] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0684] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0685] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0686] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0687] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0688] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0689] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0690] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0691] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0692] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0693] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0694] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0695] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[0696] Device behavior

[0697] The terminal first starts operating when a user starts a remote conference. A camera is connected to the terminal, and video data of the user and the other party is acquired in real time using this camera. The acquired video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[0698] Server Operation

[0699] The server receives the video data sent from the device and instantly analyzes it using generative AI. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, and uses these to infer the other person's emotions and psychological state. Specifically, the generative AI identifies states such as happiness, tension, and concentration, and generates a communication method based on these. The analysis results are stored in a database and made available for immediate use.

[0700] Creation and presentation of communication methods

[0701] The server generates several effective communication methods based on the analysis results. The generated communication methods include multiple patterns, such as light-hearted conversations when the other person is relaxed, or reassuring words when the other person is nervous.

[0702] The generated communication methods are sent to the terminal and presented on the user's UI. The user can view the options and select the most suitable communication method. The user's choice from the displayed options is immediately applied, and the conversation continues.

[0703] Specific examples

[0704] For example, consider a case where a user is conducting a remote job interview. In this case, the device captures video data of the interviewer via a camera and sends it to a server. The server uses generative AI to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier." These options are presented to the user via the device, allowing them to choose the appropriate method and smoothly progress through the interview.

[0705] In this way, the present invention makes it possible to achieve effective communication in remote conferences that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[0706] The processing flow will be explained below.

[0707] Step 1:

[0708] A user starts a remote conference. The user uses a device (such as a PC or smartphone) to join the ZOOM conference using a camera and microphone. The user joins the conference URL on the ZOOM application or starts a new conference.

[0709] Step 2:

[0710] The device captures video data in real time using a camera, encodes the captured video data frame by frame, and temporarily stores it in a buffer. This video data is then sent to a server at regular intervals.

[0711] Step 3:

[0712] The server analyzes the received video data. The video data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. As a result of the analysis, the other person's emotions and psychological state are inferred.

[0713] Step 4:

[0714] The server generates several communication methods based on the analysis results. For example, if the person smiles a lot, it will judge them to be "relaxed," and if they have furrowed brows, it will judge them to be "tense," and generate a communication method appropriate to each.

[0715] Step 5:

[0716] The server generates a communication method and sends it to the device in real time, including multiple suggested patterns (e.g., humorous conversation, words of encouragement, etc.).

[0717] Step 6:

[0718] The device then presents the received communication method to the user. The user selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option. This selection allows for smooth communication that is close to face-to-face.

[0719] Step 7:

[0720] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[0721] In this way, the system of the present invention can effectively obtain non-verbal information in remote conferences similar to that in face-to-face conferences and provide appropriate means of communication, thereby improving the quality of dialogue.

[0722] Example 1

[0723] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0724] In remote conferences, non-face-to-face communication between participants poses the challenge of accurately predicting the other person's emotions and psychological state from their facial expressions and gestures, and using appropriate communication methods. As a result, the effectiveness of communication decreases, and the possibility of misunderstandings and breakdowns in dialogue increases.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0726] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to a data processing device, and means for the data processing device to analyze the video data. This enables the user to analyze the emotions and psychological state of the other person and generate and present an effective communication method based on that analysis.

[0727] "Video data" refers to visual information captured using a camera or other optical device.

[0728] A "data processing device" is a device for analyzing video data and processing the information using specific algorithms.

[0729] "Analyzing" is the process of using captured video data to detect and evaluate specific patterns or characteristics.

[0730] "Communication methods" refer to the means and approaches of dialogue that are generated based on the emotions and psychological state of the other person.

[0731] A "display device" is a device for visually presenting the generated communication method to the user.

[0732] "User" refers to an individual or group that operates this system and interacts with it through a terminal.

[0733] A "means" is a device, process, or system element designed to accomplish a particular purpose.

[0734] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[0735] Device behavior

[0736] The device first begins operation when a user starts a remote conference. The specific hardware used is a camera (e.g., a standard webcam). This camera is used to capture video data of the user and the other party in real time. The captured video data is captured at a rate of 30 frames per second and compressed and encoded using FFmpeg software. The encoded video data is then sent to the server using WebSocket.

[0737] Server Operation

[0738] The server receives the video data sent from the device and immediately begins analysis. For analysis, a generative AI model (e.g., a general generative AI model and TensorFlow) is used. This generative AI extracts features such as facial expressions, gestures, complexion, and gaze from the video data, and infers psychological states such as happiness, tension, and concentration. The analysis results are stored in a database (e.g., a general relational database or NoSQL database).

[0739] Creation and presentation of communication methods

[0740] The server generates several effective communication methods based on the analysis results. This generation uses a generative AI model (e.g., a general generative AI model) with natural language generation (NLG) capabilities. The generated communication methods include multiple patterns, such as light-hearted banter when the other person is relaxed, or reassuring words when the other person is nervous. Below is an example of a prompt:

[0741] Sample prompt: "Generate appropriate communication suggestions if the other person is nervous."

[0742] The generated communication method is converted into JSON format and sent to the terminal via WebSocket communication.

[0743] Display on the device's user interface

[0744] The device presents the received communication method on a user interface (UI), for example, displaying options such as "Recommend deep breathing to relax" or "Start with a simple question to make it easier to talk."

[0745] User selection and application

[0746] The user selects the communication method that they think is most appropriate from the options displayed on the device's UI. The selected method is immediately applied and the conversation continues.

[0747] Specific examples

[0748] For example, when a user conducts a remote job interview, the device uses a camera to capture video data of the interviewer and sends it to a server. The server uses a generative AI model to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier to understand," and sends these to the device. The device then presents these options on a user interface, allowing the user to select the appropriate method to smoothly proceed with the interview.

[0749] Through this procedure, the present invention enables effective communication in remote meetings that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[0750] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0751] Step 1:

[0752] A user starts the meeting software on a terminal and starts a remote conference. The input is the user's operation (a conference start action), and the output is the system entering an operational state.

[0753] Step 2:

[0754] The device uses a camera to capture real-time video data of the user and the target. The input is the video signal from the camera, and the output is the captured video data. FFmpeg is used to compress and encode the video data.

[0755] Step 3:

[0756] The device sends the encoded video data to the server using WebSocket communication. The input is the encoded video data, and the output is data packets via WebSocket.

[0757] Step 4:

[0758] The server receives video data sent from the device. The input is data packets sent over WebSocket, and the output is parseable video data, which the server stores in a buffer.

[0759] Step 5:

[0760] The server analyzes the video data using a generative AI model. The input is the received video data, and the output is characteristic data such as facial expressions, gestures, complexion, and gaze. Based on this data, the server processes the other person's emotions and psychological state.

[0761] Step 6:

[0762] The server generates several effective communication methods based on the analysis results. The input is the analysis results (emotions and psychological state), and the output is the generated multiple communication methods. A generative AI model with natural language generation (NLG) capabilities is used for generation.

[0763] Step 7:

[0764] The server converts the generated communication methods into JSON format and sends it to the terminal via WebSocket communication. The input is multiple communication methods (text data), and the output is JSON data, which is sent to the terminal via WebSocket.

[0765] Step 8:

[0766] The device parses the received JSON data and displays the generated communication methods on the user interface (UI). The input is JSON-formatted data, and the output is options on the UI. For example, options such as "Recommend deep breathing" or "Ask a simple question" are displayed.

[0767] Step 9:

[0768] The user selects the most appropriate communication method from the displayed options. The input is the UI option, and the output is the selected communication method. The selected method is applied immediately.

[0769] Step 10:

[0770] The device reports the selected communication method to the server and the dialogue continues. The input is the communication method selected by the user, and the output is the selection sent to the server, which uses this to prepare for the next analysis cycle.

[0771] (Application example 1)

[0772] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0773] In industrial settings, there is a need to grasp the emotions and psychological state of workers in real time and provide appropriate feedback accordingly. However, current systems have difficulty in doing so effectively, which can lead to a decline in worker efficiency and performance. In addition, there is a lack of means to provide instructions tailored to specific situations in real time, resulting in a lack of immediate response capabilities on-site. This poses challenges for improving worker safety and work efficiency.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0775] In this invention, the server includes means for analyzing video data, means for generating a communication method based on the analysis results, means for presenting the generated communication method to the terminal, means for creating and presenting feedback in real time based on the analysis results, and means for providing appropriate instructions according to the worker's situation. This makes it possible to accurately grasp the worker's emotions and psychological state and provide appropriate feedback and instructions based on that in real time.

[0776] "Video data" refers to video signals and image information acquired through devices such as cameras.

[0777] "Means for acquiring" refers to a method or device for acquiring video data using a device such as a camera.

[0778] "Server" refers to a central computing device for storing, processing, and analyzing data over a network.

[0779] "Means of analysis" refers to methods and software for extracting, evaluating, and determining specific information using acquired data.

[0780] "Means for generating communication methods" refers to systems or processes for creating appropriate dialogue methods or instructions based on the analysis results.

[0781] "Terminal" refers to a device or display device that can be directly operated by a user.

[0782] "Presenting means" refers to a method or device for visually or audibly presenting the generated information to the user.

[0783] "User" refers to a person or operator who uses the system.

[0784] "Feedback" refers to information such as evaluations, instructions, and advice provided to workers by the system.

[0785] "Appropriate instructions" refers to specific guidance and advice on the work content that is provided immediately based on the analysis results and in accordance with the situation at hand and the worker's condition.

[0786] To implement this invention, a camera for acquiring video data, a server for processing and storing the data, and a terminal for providing feedback to the user must all work together. The specific operation of this system will now be described.

[0787] Device behavior

[0788] The terminal first starts operation when the user starts. A camera is connected to the terminal, and this camera is used to capture video data of the user and workers in real time. The captured video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[0789] Server Operation

[0790] The server receives the video data sent from the device and instantly analyzes it using a generative AI model. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, to infer the worker's emotions and psychological state. Specifically, the generative AI model identifies states such as happiness, tension, and concentration, and generates work instructions and feedback based on these. The analysis results are stored in a database and made available for immediate use.

[0791] Generating and Presenting Feedback

[0792] The server generates effective feedback based on the analysis results. This generated feedback includes multiple patterns, such as encouraging the worker to "take a slow, deep breath" if they are feeling nervous, or instructions to "take your time and work carefully." The generated feedback is sent to the device and displayed on the user's UI. The user can view this and select the most appropriate feedback. The user's choice from the displayed options is immediately applied, and the work continues.

[0793] Specific examples

[0794] For example, if a worker is operating a new machine on a factory production line, the device captures the worker's video data via a camera and sends it to a server. The server then analyzes the video data using a generative AI model and determines that the worker is nervous. Based on this, the server generates feedback such as "Take a deep breath and relax" or "Please work slowly." These options are presented to the user via the device, allowing them to choose the appropriate method to support the worker.

[0795] Prompt Sentence Examples

[0796] Here are some example prompts to input to a generative AI model:

[0797] Analyze the worker's facial expressions, gestures, complexion, and gaze from the input image to infer the worker's emotions and psychological state. Then, generate the following feedback: reassuring words if the worker is nervous, light-hearted conversation if the worker is relaxed, and encouraging words to keep working if the worker is concentrating.

[0798] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0799] Step 1:

[0800] The device uses a camera to capture video data in real time.

[0801] The input is the live video from the camera, and the output is the captured video data. Specifically, the camera sensor captures the light signal and converts it into digital video data.

[0802] Step 2:

[0803] The video data acquired by the terminal is sent to the server at regular intervals.

[0804] The input is the captured video data, and the output is the data sent to the server. Specifically, the operation is to use a network protocol to configure the video data into packets and send them to the server.

[0805] Step 3:

[0806] The video data received by the server is analyzed using a generative AI model.

[0807] The input is video data sent from the device, and the output is analyzed data on emotions and psychological states. Specifically, the system analyzes the video data frame by frame, identifies facial expressions, gestures, complexion, and gaze, and processes the data to determine emotions and feelings.

[0808] Step 4:

[0809] The server generates feedback based on the analysis results.

[0810] The input is the analyzed emotional and psychological data, and the output is the generated feedback. Specifically, the analysis results are used as metadata, and prompt sentences are input to the generative AI model to generate appropriate feedback.

[0811] Step 5:

[0812] The server transmits the generated feedback to the terminal.

[0813] The input is the generated feedback, and the output is the feedback sent to the terminal. The specific operation is to configure the feedback as a packet via a network protocol and send it to the terminal.

[0814] Step 6:

[0815] The device presents the received feedback on the user's UI.

[0816] The input is the feedback sent from the server, and the output is the feedback presented to the user. The specific operation is to process the received data and display it in an appropriate format on the user interface.

[0817] Step 7:

[0818] The user selects the feedback provided.

[0819] The input is multiple feedback options presented on the UI, and the output is the selected feedback. The specific operation is to accept the user's operation, record the selection result, and save it in a form that can be used for subsequent processing.

[0820] Step 8:

[0821] Provide the worker with user-selected feedback.

[0822] The input is the selected feedback, and the output is the feedback provided to the worker, which can be communicated to the worker via a terminal or smart glasses in the form of voice or text.

[0823] This series of processing steps enables the system to provide appropriate feedback in real time according to the worker's emotions and psychological state.

[0824] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0825] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0826] Device behavior

[0827] The device starts operating when a user starts a remote conference. A camera and microphone are connected to the device, and they are used to capture video data of the user and the other party in real time. The captured video data is encoded frame by frame and temporarily stored in a buffer. This video data is then sent to the server at regular intervals.

[0828] Server Operation

[0829] The server receives the video data sent from the device and immediately begins analysis. This analysis uses generative AI and an emotion engine. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze. The emotion engine uses this data to recognize the user's emotions in real time.

[0830] Emotion Engine

[0831] The emotion engine uses machine learning algorithms to accurately determine a user's emotions. It combines facial expression analysis, gesture analysis, facial color analysis, and eye gaze analysis to identify states such as happiness, tension, and concentration. The results are immediately stored on the server and used as analysis results.

[0832] Creation and presentation of communication methods

[0833] The server generates multiple effective communication methods based on the analysis results of the emotion engine, such as light-hearted conversation if the user is relaxed, or reassuring words if the user is nervous.

[0834] The generated communication methods are sent to the device in real time and displayed on the user's UI. The user can select the appropriate option from these options and proceed with the conversation accordingly. This selection enables smooth communication that is close to face-to-face.

[0835] Specific examples

[0836] For example, consider a case where a user is engaged in an online counseling session. In this case, the device captures video data of the counseling session via a camera and sends it to a server. The server analyzes the video data using generative AI and an emotion engine to determine that the client is nervous. Based on this determination, the server generates communication methods such as "I'll teach you breathing techniques to help you relax" or "You can speak more slowly." These options are presented to the user via the device, allowing them to choose the most appropriate method and smoothly progress with the counseling session.

[0837] Convenience

[0838] This system can be applied in a variety of situations, including in education, corporate interviews, and counseling. By providing real-time feedback based on analysis of non-verbal information and providing effective communication methods, it is possible to achieve high-quality dialogue similar to face-to-face, even in a remote environment.

[0839] In this way, the present invention can significantly improve the quality of remote communication and facilitate smooth dialogue.

[0840] The processing flow will be explained below.

[0841] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0842] Step 1:

[0843] A user starts a remote conference. Using a device (PC, smartphone, etc.), the user joins the conference URL on the ZOOM application or starts a new conference. At this time, make sure that the camera and microphone are connected correctly.

[0844] Step 2:

[0845] The device uses a camera to acquire video data in real time. The device uses the camera device to capture and encode video data for each frame. The encoded video data is temporarily stored in a buffer.

[0846] Step 3:

[0847] The video data acquired by the device is sent to a server at regular intervals. The video data is then uploaded to the server via the Internet. A dedicated API and data transmission protocol are used for this purpose.

[0848] Step 4:

[0849] The server analyzes the received video data. The received data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. This analysis allows features to be extracted from the data.

[0850] Step 5:

[0851] The server uses an emotion engine to recognize the user's emotions in real time. The emotion engine uses a machine learning algorithm based on the analyzed features to determine emotions such as happiness, tension, and concentration. The results of the determination are stored internally on the server.

[0852] Step 6:

[0853] The server generates multiple communication methods based on the results of the emotion engine. For example, if the user is relaxed, it generates methods such as "continue the conversation in a relaxed manner," and if the user is nervous, it generates methods such as "say something comforting."

[0854] Step 7:

[0855] The server sends the generated communication methods to the device. The sent data includes multiple suggested patterns (light jokes, words of encouragement, etc.). These communication methods are sent in real time.

[0856] Step 8:

[0857] The device then presents the received communication method to the user, who then selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option.

[0858] Step 9:

[0859] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[0860] In this way, the system of the present invention can effectively acquire non-verbal information in remote conferences just as it does in face-to-face conferences, and can improve the quality of dialogue by providing appropriate means of communication.

[0861] Example 2

[0862] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0863] In remote meetings and interviews, it is difficult to accurately grasp the user's emotions and psychological state and communicate effectively. In addition, there is a lack of systems that can analyze non-verbal information and provide appropriate communication methods in real time.

[0864] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0865] In this invention, the server includes means for acquiring visual and audio data from the user, means for transmitting the acquired visual and audio data to the server, means for the server to analyze the visual and audio data, means for generating appropriate communication means based on the analysis results, means for presenting the generated communication means to the terminal, and means for the user to select the presented communication means. This makes it possible to analyze the user's emotions and psychological state in real time and provide appropriate communication means.

[0866] "Visual data" is data in digital form that contains images of the user and their counterparts.

[0867] "Voice data" is audio data that includes the pronunciation and conversation content of the user and the other party.

[0868] "Means for acquiring" refers to devices and methods for acquiring data using a camera and microphone.

[0869] "Means for sending to the server" refers to the communication protocol and infrastructure for transferring the acquired data to the server.

[0870] "Means for analyzing" means software and hardware for processing and analyzing acquired visual and audio data.

[0871] "Means for generating appropriate communication methods" are algorithms or models that suggest effective conversations and actions based on the analysis results.

[0872] The "means for presenting on the terminal" is an interface for visually displaying the generated communication means to the user.

[0873] "User-selected means" refers to an interface that allows the user to select and execute an appropriate communication means from the presented means.

[0874] An "expression" is a visual expression of emotion expressed by the movement of specific muscles in the face.

[0875] "Movement" is behavior data that indicates the user's physical movements and posture.

[0876] "Skin tone" is data that represents changes in the complexion and skin color of the user.

[0877] "Gaze" is data that includes the user's eye movements and gaze points.

[0878] MODE FOR CARRYING OUT THE INVENTION

[0879] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[0880] Device behavior

[0881] The device starts operating when a user starts a remote conference. The device is equipped with a camera and microphone, which are used to capture video data of the user and the other party in real time. The captured video data is processed frame by frame using encoding methods such as MJPEG or H.264 and temporarily stored in a buffer. This processing is often performed using OpenCV.

[0882] Example: When a user is using a web conferencing tool (e.g., an online meeting system), the camera and microphone are activated to capture video and audio. This video data is captured at 30 frames per second and stored in a buffer.

[0883] Sending data to the server

[0884] The device sends the encoded video data to the server at regular intervals using protocols such as WebSocket or RTMP (Real-Time Messaging Protocol), which allows data to be transferred with low latency.

[0885] Example: Video data captured on a device is sent to a server once per second. The protocol used is WebSocket, which enables real-time transmission.

[0886] Server-side data analysis

[0887] The server receives the video data sent from the device and immediately begins analysis. Server-side processing uses image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch.

[0888] Example: The server analyzes the frames of video data received using OpenCV to extract features such as gestures and facial expressions. These features are then further analyzed using a TensorFlow model to obtain more detailed information.

[0889] Emotion recognition by emotion engine

[0890] The server uses generative AI and an emotion engine to analyze the user's emotions from video data in real time. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze, and the emotion engine recognizes the user's emotional state based on that data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests.

[0891] Example: Facial expression analysis can quantify humorous reactions and levels of tension. For example, if a significant frown or lip twist is detected, that information is sent to an emotion engine, which then determines the emotional state, such as "tension" or "anxiety."

[0892] Creating communication methods

[0893] Based on the analysis results from the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods. Depending on the analysis results, it generates conversations that incorporate light-hearted jokes if the user is relaxed, or words that reassure the user if they are nervous.

[0894] Example: If a user is determined to be nervous, GPT-3 will generate advice such as "Take a deep breath and you'll feel better."

[0895] Example prompt sentence:

[0896] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[0897] Providing communication methods

[0898] The server sends the generated communication method to the device in real time and displays it on the user's UI via a REST API or notification service.

[0899] Example: The generated advice is displayed in a pop-up format in the upper right corner of the screen, making it easy for users to see.

[0900] User selection and execution

[0901] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that, enabling smooth communication that is close to face-to-face.

[0902] Example: A user selects the advice "Take a deep breath and you'll feel better" and verbally repeats it to the other person, making the conversation flow more naturally and effectively.

[0903] This system analyzes emotions in real time and provides effective communication methods, enabling high-quality dialogue even in remote environments. This process makes remote meetings and interviews as smooth as face-to-face meetings.

[0904] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0905] Step 1:

[0906] When a user starts a remote conference, the device activates the camera and microphone and acquires visual and audio data in real time. The device uses OpenCV to capture video data frame by frame (e.g., 30 frames per second) and stores audio data in a buffer. The acquired visual and audio data are input.

[0907] Specific operation: The camera captures the user's face, the microphone collects the user's speech, and the data is temporarily stored in the device's memory.

[0908] Input: User's visual and audio data

[0909] Output: Buffered encoded visual and audio data

[0910] Step 2:

[0911] The device transmits the visual and audio data stored in the buffer to the server at regular intervals (e.g., once per second) using a low-latency protocol such as WebSocket or RTMP.

[0912] Specific operation: The device opens a WebSocket and sends video frames and audio samples as binary data to the server.

[0913] Input: Buffered encoded visual and audio data

[0914] Output: Visual and audio data sent to the server

[0915] Step 3:

[0916] The server analyzes the visual and audio data received from the device. Image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch are used for server-side processing. The received data is first decoded and then analyzed to extract visual and audio features.

[0917] Specific operation: The server decodes and decomposes the received data into frames, and uses OpenCV to extract features such as face and eye direction. At the same time, it also analyzes the features of the audio data.

[0918] Input: Visual and audio data sent to the server

[0919] Output: Analyzed visual and audio feature data

[0920] Step 4:

[0921] The server uses generative AI and an emotion engine to recognize the user's emotions in real time from the analyzed visual and audio feature data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests to determine emotional states such as tension, joy, and concentration.

[0922] Specific operation: The emotion engine uses a neural network to analyze visual data such as facial expressions, gestures, skin tone, and gaze, and then combines this data with a machine learning algorithm to determine emotions.

[0923] Input: Analyzed visual and audio feature data

[0924] Output: Determined user's emotional state

[0925] Step 5:

[0926] Based on the analysis results of the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods, such as light-hearted banter when the user is relaxed, or reassuring words when the user is nervous.

[0927] Specific operation: A generative AI model such as GPT-3 generates appropriate dialogue content using prompt sentences based on the results of sentiment analysis.

[0928] Example prompt sentence:

[0929] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[0930] Input: Determined emotional state of the user

[0931] Output: The generated means of communication

[0932] Step 6:

[0933] The server sends the generated communication methods to the device in real time and displays them on the user's UI. This transmission uses a REST API or notification service.

[0934] Specific operation: The server converts the generated communication method into JSON format, sends it to the terminal via an HTTP request, and presents it on the terminal in a pop-up or dialog format.

[0935] Input: Generated communication method

[0936] Output: Communication options displayed on the device

[0937] Step 7:

[0938] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that choice, achieving smooth communication that is close to face-to-face.

[0939] Specific behavior: The user clicks or taps on an option presented in the UI and continues the conversation with the other person using that method.

[0940] Input: Communication options displayed on the device

[0941] Output: Dialogue content based on the communication method selected by the user

[0942] (Application example 2)

[0943] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0944] In traditional remote classes, it was difficult for teachers to instantly grasp students' emotional states and provide effective feedback at the appropriate time. Furthermore, the inability to provide appropriate explanations tailored to students' situations could result in a decline in educational effectiveness. This resulted in the quality of remote classes being inferior to face-to-face classes.

[0945] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0946] In this invention, the server includes: [means for transmitting acquired video data to the server;] [means for the server to analyze the video data;] [means for generating a communication method based on the analysis results;] [means for using smart glasses in an educational setting to present the generated communication method to a teacher in real time as feedback according to the student's emotional state; and [means for generating prompt sentences according to the emotional state and providing effective feedback.] This enables teachers to grasp students' emotional states in real time and provide appropriate feedback and supplementary explanations.

[0947] "Video data" refers to information in video frames captured using imaging equipment such as a camera.

[0948] A "server" is a computer system that processes and analyzes data over a network.

[0949] "Analysis" is the act of breaking down the information from the acquired data and examining its contents in detail.

[0950] "Generation" is the act of creating new results or means based on data or information.

[0951] A "terminal" is a hardware device that is directly operated by a user.

[0952] A "communication method" is a means or approach for effective dialogue or communication.

[0953] "Smart glasses" are glasses-type devices equipped with a camera and a display that can provide visual information.

[0954] "Feedback" is the act of returning information for the purpose of improvement or correction based on the information or results obtained.

[0955] A "prompt sentence" is an input sentence that is used to provide guidance and help to the generative AI model.

[0956] The "educational setting" refers to the place or environment where lessons and learning activities take place.

[0957] A "user" is a person or entity that uses a system or device.

[0958] "Emotional state" refers to the user's psychological state and fluctuations in feelings.

[0959] A system for specifically implementing this invention uses smart glasses, a camera, a server, a generative AI model, and an emotion engine, and the following embodiment is described in detail below.

[0960] First, the smart glasses are equipped with a camera that captures students' facial expressions, gestures, and gaze in real time during remote classes. The captured video data is temporarily stored in a buffer inside the smart glasses and then periodically sent to a server.

[0961] The server analyzes the received video data. The analysis includes a generative AI model and an emotion engine that recognizes the student's emotional state in real time based on facial expressions, gestures, complexion, gaze, and other factors. The generative AI model and emotion engine can accurately determine whether the student is in an emotional state such as happiness, confusion, or concentration.

[0962] Next, based on the analysis results, the server generates multiple effective communication methods. These methods are presented in real time to the teacher wearing the smart glasses. For example, if a student is confused, the teacher will provide feedback such as "Please explain again using easy-to-understand examples and metaphors."

[0963] Teachers can refer to these provided feedback and communication methods as they conduct their lessons, enabling them to provide feedback at the appropriate time based on students' level of understanding and emotional state, thereby improving the effectiveness of remote lessons.

[0964] Hardware and software used

[0965] Hardware: Smart glasses (with camera)

[0966] Software: OpenCV, EmotionRecognition library, CommunicationSuggestions module

[0967] Specific examples

[0968] For example, if the emotion engine recognizes that a student is confused during a lesson, the server will provide the teacher with feedback such as, "If the student is confused, please explain again using easy-to-understand examples and metaphors." The teacher can then use this feedback to re-explain the problem using concrete examples, such as, "This math problem might be easier to understand if you explain it using an apple."

[0969] Prompt Sentence Examples

[0970] "If students are confused, explain it again with an alternative, clearer example."

[0971] "If students are focused, keep them on pace."

[0972] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0973] Step 1:

[0974] The camera in the smart glasses captures video data of the students.

[0975] (Input) Real-time video of the student.

[0976] (Data processing) The camera sensor captures video frames and temporarily stores them in a buffer.

[0977] (Output) Video data for each frame.

[0978] Step 2:

[0979] The video data acquired by the smart glasses is sent to the server.

[0980] (Input) Video data stored in the buffer.

[0981] (Data processing) Video data is sent to the server reliably using network protocols.

[0982] (Output) Video data received by the server.

[0983] Step 3:

[0984] The server analyzes the received video data using a generative AI model and emotion engine.

[0985] (Input) Video data received by the server.

[0986] (Data processing) The generative AI model analyzes facial expressions, gestures, complexion, gaze, etc. for each frame, and the emotion engine uses this information to infer the student's emotional state.

[0987] (Output) Inferred emotional state (e.g., happy, confused, focused, etc.).

[0988] Step 4:

[0989] The server generates a plurality of effective communication methods based on the inferred emotional state.

[0990] (Input) Inferred emotional state.

[0991] (Data processing) Generate prompt sentences that correspond to the emotional state, and generate effective feedback methods using a generative AI model.

[0992] (Output) Multiple communication methods generated.

[0993] Step 5:

[0994] The server presents the generated communication method to the smart glasses in real time.

[0995] (Input) Multiple communication methods generated.

[0996] (Data processing) Displayed as text data on the smart glasses display.

[0997] (Output) Communication methods displayed on the smart glasses display.

[0998] Step 6:

[0999] The user, the teacher, selects the communication method presented and applies it to their lessons.

[1000] (Input) Communication methods displayed on the smart glasses display.

[1001] (Data processing) Teachers can choose to provide feedback at appropriate times according to students' emotional state.

[1002] (Output) Appropriate feedback from the teacher to the student.

[1003] Examples:

[1004] Example prompt sentence:

[1005] "If students are confused, explain it again with an alternative, clearer example."

[1006] "If students are focused, keep them on pace."

[1007] In this way, the system operates while clearly defining the specific input, data processing, and output at each step, allowing teachers to provide appropriate feedback to students in real time during remote classes based on their emotional state.

[1008] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1009] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1010] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1011] [Fourth embodiment]

[1012] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1013] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1014] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1015] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1016] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1017] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1018] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1019] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1020] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1021] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1022] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1023] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1024] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1025] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[1026] Device behavior

[1027] The terminal first starts operating when a user starts a remote conference. A camera is connected to the terminal, and video data of the user and the other party is acquired in real time using this camera. The acquired video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[1028] Server Operation

[1029] The server receives the video data sent from the device and instantly analyzes it using generative AI. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, and uses these to infer the other person's emotions and psychological state. Specifically, the generative AI identifies states such as happiness, tension, and concentration, and generates a communication method based on these. The analysis results are stored in a database and made available for immediate use.

[1030] Creation and presentation of communication methods

[1031] The server generates several effective communication methods based on the analysis results. The generated communication methods include multiple patterns, such as light-hearted conversations when the other person is relaxed, or reassuring words when the other person is nervous.

[1032] The generated communication methods are sent to the terminal and presented on the user's UI. The user can view the options and select the most suitable communication method. The user's choice from the displayed options is immediately applied, and the conversation continues.

[1033] Specific examples

[1034] For example, consider a case where a user is conducting a remote job interview. In this case, the device captures video data of the interviewer via a camera and sends it to a server. The server uses generative AI to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier." These options are presented to the user via the device, allowing them to choose the appropriate method and smoothly progress through the interview.

[1035] In this way, the present invention makes it possible to achieve effective communication in remote conferences that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[1036] The processing flow will be explained below.

[1037] Step 1:

[1038] A user starts a remote conference. The user uses a device (such as a PC or smartphone) to join the ZOOM conference using a camera and microphone. The user joins the conference URL on the ZOOM application or starts a new conference.

[1039] Step 2:

[1040] The device captures video data in real time using a camera, encodes the captured video data frame by frame, and temporarily stores it in a buffer. This video data is then sent to a server at regular intervals.

[1041] Step 3:

[1042] The server analyzes the received video data. The video data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. As a result of the analysis, the other person's emotions and psychological state are inferred.

[1043] Step 4:

[1044] The server generates several communication methods based on the analysis results. For example, if the person smiles a lot, it will judge them to be "relaxed," and if they have furrowed brows, it will judge them to be "tense," and generate a communication method appropriate to each.

[1045] Step 5:

[1046] The server generates a communication method and sends it to the device in real time, including multiple suggested patterns (e.g., humorous conversation, words of encouragement, etc.).

[1047] Step 6:

[1048] The device then presents the received communication method to the user. The user selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option. This selection allows for smooth communication that is close to face-to-face.

[1049] Step 7:

[1050] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[1051] In this way, the system of the present invention can effectively obtain non-verbal information in remote conferences similar to that in face-to-face conferences and provide appropriate means of communication, thereby improving the quality of dialogue.

[1052] Example 1

[1053] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1054] In remote conferences, non-face-to-face communication between participants poses the challenge of accurately predicting the other person's emotions and psychological state from their facial expressions and gestures, and using appropriate communication methods. As a result, the effectiveness of communication decreases, and the possibility of misunderstandings and breakdowns in dialogue increases.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1056] In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to a data processing device, and means for the data processing device to analyze the video data. This enables the user to analyze the emotions and psychological state of the other person and generate and present an effective communication method based on that analysis.

[1057] "Video data" refers to visual information captured using a camera or other optical device.

[1058] A "data processing device" is a device for analyzing video data and processing the information using specific algorithms.

[1059] "Analyzing" is the process of using captured video data to detect and evaluate specific patterns or characteristics.

[1060] "Communication methods" refer to the means and approaches of dialogue that are generated based on the emotions and psychological state of the other person.

[1061] A "display device" is a device for visually presenting the generated communication method to the user.

[1062] "User" refers to an individual or group that operates this system and interacts with it through a terminal.

[1063] A "means" is a device, process, or system element designed to accomplish a particular purpose.

[1064] In order to implement the present invention, the terminal, the server, and the user must work in cooperation with each other. A specific embodiment of the use of this system will be described below.

[1065] Device behavior

[1066] The device first begins operation when a user starts a remote conference. The specific hardware used is a camera (e.g., a standard webcam). This camera is used to capture video data of the user and the other party in real time. The captured video data is captured at a rate of 30 frames per second and compressed and encoded using FFmpeg software. The encoded video data is then sent to the server using WebSocket.

[1067] Server Operation

[1068] The server receives the video data sent from the device and immediately begins analysis. For analysis, a generative AI model (e.g., a general generative AI model and TensorFlow) is used. This generative AI extracts features such as facial expressions, gestures, complexion, and gaze from the video data, and infers psychological states such as happiness, tension, and concentration. The analysis results are stored in a database (e.g., a general relational database or NoSQL database).

[1069] Creation and presentation of communication methods

[1070] The server generates several effective communication methods based on the analysis results. This generation uses a generative AI model (e.g., a general generative AI model) with natural language generation (NLG) capabilities. The generated communication methods include multiple patterns, such as light-hearted banter when the other person is relaxed, or reassuring words when the other person is nervous. Below is an example of a prompt:

[1071] Sample prompt: "Generate appropriate communication suggestions if the other person is nervous."

[1072] The generated communication method is converted into JSON format and sent to the terminal via WebSocket communication.

[1073] Display on the device's user interface

[1074] The device presents the received communication method on a user interface (UI), for example, displaying options such as "Recommend deep breathing to relax" or "Start with a simple question to make it easier to talk."

[1075] User selection and application

[1076] The user selects the communication method that they think is most appropriate from the options displayed on the device's UI. The selected method is immediately applied and the conversation continues.

[1077] Specific examples

[1078] For example, when a user conducts a remote job interview, the device uses a camera to capture video data of the interviewer and sends it to a server. The server uses a generative AI model to analyze the video data and determine that the interviewer is nervous. Based on this, the server generates communication methods such as "encourage them to take a deep breath to relax" or "start with simple questions to make the interview easier to understand," and sends these to the device. The device then presents these options on a user interface, allowing the user to select the appropriate method to smoothly proceed with the interview.

[1079] Through this procedure, the present invention enables effective communication in remote meetings that is similar to face-to-face meetings. This system can also be applied in a variety of settings, such as education and counseling.

[1080] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1081] Step 1:

[1082] A user starts the meeting software on a terminal and starts a remote conference. The input is the user's operation (a conference start action), and the output is the system entering an operational state.

[1083] Step 2:

[1084] The device uses a camera to capture real-time video data of the user and the target. The input is the video signal from the camera, and the output is the captured video data. FFmpeg is used to compress and encode the video data.

[1085] Step 3:

[1086] The device sends the encoded video data to the server using WebSocket communication. The input is the encoded video data, and the output is data packets via WebSocket.

[1087] Step 4:

[1088] The server receives video data sent from the device. The input is data packets sent over WebSocket, and the output is parseable video data, which the server stores in a buffer.

[1089] Step 5:

[1090] The server analyzes the video data using a generative AI model. The input is the received video data, and the output is characteristic data such as facial expressions, gestures, complexion, and gaze. Based on this data, the server processes the other person's emotions and psychological state.

[1091] Step 6:

[1092] The server generates several effective communication methods based on the analysis results. The input is the analysis results (emotions and psychological state), and the output is the generated multiple communication methods. A generative AI model with natural language generation (NLG) capabilities is used for generation.

[1093] Step 7:

[1094] The server converts the generated communication methods into JSON format and sends it to the terminal via WebSocket communication. The input is multiple communication methods (text data), and the output is JSON data, which is sent to the terminal via WebSocket.

[1095] Step 8:

[1096] The device parses the received JSON data and displays the generated communication methods on the user interface (UI). The input is JSON-formatted data, and the output is options on the UI. For example, options such as "Recommend deep breathing" or "Ask a simple question" are displayed.

[1097] Step 9:

[1098] The user selects the most appropriate communication method from the displayed options. The input is the UI option, and the output is the selected communication method. The selected method is applied immediately.

[1099] Step 10:

[1100] The device reports the selected communication method to the server and the dialogue continues. The input is the communication method selected by the user, and the output is the selection sent to the server, which uses this to prepare for the next analysis cycle.

[1101] (Application example 1)

[1102] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1103] In industrial settings, there is a need to grasp the emotions and psychological state of workers in real time and provide appropriate feedback accordingly. However, current systems have difficulty in doing so effectively, which can lead to a decline in worker efficiency and performance. In addition, there is a lack of means to provide instructions tailored to specific situations in real time, resulting in a lack of immediate response capabilities on-site. This poses challenges for improving worker safety and work efficiency.

[1104] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1105] In this invention, the server includes means for analyzing video data, means for generating a communication method based on the analysis results, means for presenting the generated communication method to the terminal, means for creating and presenting feedback in real time based on the analysis results, and means for providing appropriate instructions according to the worker's situation. This makes it possible to accurately grasp the worker's emotions and psychological state and provide appropriate feedback and instructions based on that in real time.

[1106] "Video data" refers to video signals and image information acquired through devices such as cameras.

[1107] "Means for acquiring" refers to a method or device for acquiring video data using a device such as a camera.

[1108] "Server" refers to a central computing device for storing, processing, and analyzing data over a network.

[1109] "Means of analysis" refers to methods and software for extracting, evaluating, and determining specific information using acquired data.

[1110] "Means for generating communication methods" refers to systems or processes for creating appropriate dialogue methods or instructions based on the analysis results.

[1111] "Terminal" refers to a device or display device that can be directly operated by a user.

[1112] "Presenting means" refers to a method or device for visually or audibly presenting the generated information to the user.

[1113] "User" refers to a person or operator who uses the system.

[1114] "Feedback" refers to information such as evaluations, instructions, and advice provided to workers by the system.

[1115] "Appropriate instructions" refers to specific guidance and advice on the work content that is provided immediately based on the analysis results and in accordance with the situation at hand and the worker's condition.

[1116] To implement this invention, a camera for acquiring video data, a server for processing and storing the data, and a terminal for providing feedback to the user must all work together. The specific operation of this system will now be described.

[1117] Device behavior

[1118] The terminal first starts operation when the user starts. A camera is connected to the terminal, and this camera is used to capture video data of the user and workers in real time. The captured video data is sent to the server at regular intervals, frame by frame. This allows the server to always receive the latest video data.

[1119] Server Operation

[1120] The server receives the video data sent from the device and instantly analyzes it using a generative AI model. The analysis targets facial expressions, gestures, complexion, and gaze, among other things, to infer the worker's emotions and psychological state. Specifically, the generative AI model identifies states such as happiness, tension, and concentration, and generates work instructions and feedback based on these. The analysis results are stored in a database and made available for immediate use.

[1121] Generating and Presenting Feedback

[1122] The server generates effective feedback based on the analysis results. This generated feedback includes multiple patterns, such as encouraging the worker to "take a slow, deep breath" if they are feeling nervous, or instructions to "take your time and work carefully." The generated feedback is sent to the device and displayed on the user's UI. The user can view this and select the most appropriate feedback. The user's choice from the displayed options is immediately applied, and the work continues.

[1123] Specific examples

[1124] For example, if a worker is operating a new machine on a factory production line, the device captures the worker's video data via a camera and sends it to a server. The server then analyzes the video data using a generative AI model and determines that the worker is nervous. Based on this, the server generates feedback such as "Take a deep breath and relax" or "Please work slowly." These options are presented to the user via the device, allowing them to choose the appropriate method to support the worker.

[1125] Prompt Sentence Examples

[1126] Here are some example prompts to input to a generative AI model:

[1127] Analyze the worker's facial expressions, gestures, complexion, and gaze from the input image to infer the worker's emotions and psychological state. Then, generate the following feedback: reassuring words if the worker is nervous, light-hearted conversation if the worker is relaxed, and encouraging words to keep working if the worker is concentrating.

[1128] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1129] Step 1:

[1130] The device uses a camera to capture video data in real time.

[1131] The input is the live video from the camera, and the output is the captured video data. Specifically, the camera sensor captures the light signal and converts it into digital video data.

[1132] Step 2:

[1133] The video data acquired by the terminal is sent to the server at regular intervals.

[1134] The input is the captured video data, and the output is the data sent to the server. Specifically, the operation is to use a network protocol to configure the video data into packets and send them to the server.

[1135] Step 3:

[1136] The video data received by the server is analyzed using a generative AI model.

[1137] The input is video data sent from the device, and the output is analyzed data on emotions and psychological states. Specifically, the system analyzes the video data frame by frame, identifies facial expressions, gestures, complexion, and gaze, and processes the data to determine emotions and feelings.

[1138] Step 4:

[1139] The server generates feedback based on the analysis results.

[1140] The input is the analyzed emotional and psychological data, and the output is the generated feedback. Specifically, the analysis results are used as metadata, and prompt sentences are input to the generative AI model to generate appropriate feedback.

[1141] Step 5:

[1142] The server transmits the generated feedback to the terminal.

[1143] The input is the generated feedback, and the output is the feedback sent to the terminal. The specific operation is to configure the feedback as a packet via a network protocol and send it to the terminal.

[1144] Step 6:

[1145] The device presents the received feedback on the user's UI.

[1146] The input is the feedback sent from the server, and the output is the feedback presented to the user. The specific operation is to process the received data and display it in an appropriate format on the user interface.

[1147] Step 7:

[1148] The user selects the feedback provided.

[1149] The input is multiple feedback options presented on the UI, and the output is the selected feedback. The specific operation is to accept the user's operation, record the selection result, and save it in a form that can be used for subsequent processing.

[1150] Step 8:

[1151] Provide the worker with user-selected feedback.

[1152] The input is the selected feedback, and the output is the feedback provided to the worker, which can be communicated to the worker via a terminal or smart glasses in the form of voice or text.

[1153] This series of processing steps enables the system to provide appropriate feedback in real time according to the worker's emotions and psychological state.

[1154] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1155] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[1156] Device behavior

[1157] The device starts operating when a user starts a remote conference. A camera and microphone are connected to the device, and they are used to capture video data of the user and the other party in real time. The captured video data is encoded frame by frame and temporarily stored in a buffer. This video data is then sent to the server at regular intervals.

[1158] Server Operation

[1159] The server receives the video data sent from the device and immediately begins analysis. This analysis uses generative AI and an emotion engine. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze. The emotion engine uses this data to recognize the user's emotions in real time.

[1160] Emotion Engine

[1161] The emotion engine uses machine learning algorithms to accurately determine a user's emotions. It combines facial expression analysis, gesture analysis, facial color analysis, and eye gaze analysis to identify states such as happiness, tension, and concentration. The results are immediately stored on the server and used as analysis results.

[1162] Creation and presentation of communication methods

[1163] The server generates multiple effective communication methods based on the analysis results of the emotion engine, such as light-hearted conversation if the user is relaxed, or reassuring words if the user is nervous.

[1164] The generated communication methods are sent to the device in real time and displayed on the user's UI. The user can select the appropriate option from these options and proceed with the conversation accordingly. This selection enables smooth communication that is close to face-to-face.

[1165] Specific examples

[1166] For example, consider a case where a user is engaged in an online counseling session. In this case, the device captures video data of the counseling session via a camera and sends it to a server. The server analyzes the video data using generative AI and an emotion engine to determine that the client is nervous. Based on this determination, the server generates communication methods such as "I'll teach you breathing techniques to help you relax" or "You can speak more slowly." These options are presented to the user via the device, allowing them to choose the most appropriate method and smoothly progress with the counseling session.

[1167] Convenience

[1168] This system can be applied in a variety of situations, including in education, corporate interviews, and counseling. By providing real-time feedback based on analysis of non-verbal information and providing effective communication methods, it is possible to achieve high-quality dialogue similar to face-to-face, even in a remote environment.

[1169] In this way, the present invention can significantly improve the quality of remote communication and facilitate smooth dialogue.

[1170] The processing flow will be explained below.

[1171] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[1172] Step 1:

[1173] A user starts a remote conference. Using a device (PC, smartphone, etc.), the user joins the conference URL on the ZOOM application or starts a new conference. At this time, make sure that the camera and microphone are connected correctly.

[1174] Step 2:

[1175] The device uses a camera to acquire video data in real time. The device uses the camera device to capture and encode video data for each frame. The encoded video data is temporarily stored in a buffer.

[1176] Step 3:

[1177] The video data acquired by the device is sent to a server at regular intervals. The video data is then uploaded to the server via the Internet. A dedicated API and data transmission protocol are used for this purpose.

[1178] Step 4:

[1179] The server analyzes the received video data. The received data is then passed to a generative AI, which analyzes elements such as facial expressions, gestures, complexion, and gaze in real time. This analysis allows features to be extracted from the data.

[1180] Step 5:

[1181] The server uses an emotion engine to recognize the user's emotions in real time. The emotion engine uses a machine learning algorithm based on the analyzed features to determine emotions such as happiness, tension, and concentration. The results of the determination are stored internally on the server.

[1182] Step 6:

[1183] The server generates multiple communication methods based on the results of the emotion engine. For example, if the user is relaxed, it generates methods such as "continue the conversation in a relaxed manner," and if the user is nervous, it generates methods such as "say something comforting."

[1184] Step 7:

[1185] The server sends the generated communication methods to the device. The sent data includes multiple suggested patterns (light jokes, words of encouragement, etc.). These communication methods are sent in real time.

[1186] Step 8:

[1187] The device then presents the received communication method to the user, who then selects the appropriate option from the multiple options displayed on the UI and proceeds with the conversation based on that option.

[1188] Step 9:

[1189] The conversation continues according to the communication method selected by the user. During this process, new video data is acquired and sent to the server again. This allows the latest analysis results and communication methods to be provided in real time.

[1190] In this way, the system of the present invention can effectively acquire non-verbal information in remote conferences just as it does in face-to-face conferences, and can improve the quality of dialogue by providing appropriate means of communication.

[1191] Example 2

[1192] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1193] In remote meetings and interviews, it is difficult to accurately grasp the user's emotions and psychological state and communicate effectively. In addition, there is a lack of systems that can analyze non-verbal information and provide appropriate communication methods in real time.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1195] In this invention, the server includes means for acquiring visual and audio data from the user, means for transmitting the acquired visual and audio data to the server, means for the server to analyze the visual and audio data, means for generating appropriate communication means based on the analysis results, means for presenting the generated communication means to the terminal, and means for the user to select the presented communication means. This makes it possible to analyze the user's emotions and psychological state in real time and provide appropriate communication means.

[1196] "Visual data" is data in digital form that contains images of the user and their counterparts.

[1197] "Voice data" is audio data that includes the pronunciation and conversation content of the user and the other party.

[1198] "Means for acquiring" refers to devices and methods for acquiring data using a camera and microphone.

[1199] "Means for sending to the server" refers to the communication protocol and infrastructure for transferring the acquired data to the server.

[1200] "Means for analyzing" means software and hardware for processing and analyzing acquired visual and audio data.

[1201] "Means for generating appropriate communication methods" are algorithms or models that suggest effective conversations and actions based on the analysis results.

[1202] The "means for presenting on the terminal" is an interface for visually displaying the generated communication means to the user.

[1203] "User-selected means" refers to an interface that allows the user to select and execute an appropriate communication means from the presented means.

[1204] An "expression" is a visual expression of emotion expressed by the movement of specific muscles in the face.

[1205] "Movement" is behavior data that indicates the user's physical movements and posture.

[1206] "Skin tone" is data that represents changes in the complexion and skin color of the user.

[1207] "Gaze" is data that includes the user's eye movements and gaze points.

[1208] MODE FOR CARRYING OUT THE INVENTION

[1209] The present invention provides a system for recognizing the emotions of users in real time during remote meetings or interviews, and providing effective communication methods that correspond to the psychological state of the other party. Specific embodiments of this system are described below.

[1210] Device behavior

[1211] The device starts operating when a user starts a remote conference. The device is equipped with a camera and microphone, which are used to capture video data of the user and the other party in real time. The captured video data is processed frame by frame using encoding methods such as MJPEG or H.264 and temporarily stored in a buffer. This processing is often performed using OpenCV.

[1212] Example: When a user is using a web conferencing tool (e.g., an online meeting system), the camera and microphone are activated to capture video and audio. This video data is captured at 30 frames per second and stored in a buffer.

[1213] Sending data to the server

[1214] The device sends the encoded video data to the server at regular intervals using protocols such as WebSocket or RTMP (Real-Time Messaging Protocol), which allows data to be transferred with low latency.

[1215] Example: Video data captured on a device is sent to a server once per second. The protocol used is WebSocket, which enables real-time transmission.

[1216] Server-side data analysis

[1217] The server receives the video data sent from the device and immediately begins analysis. Server-side processing uses image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch.

[1218] Example: The server analyzes the frames of video data received using OpenCV to extract features such as gestures and facial expressions. These features are then further analyzed using a TensorFlow model to obtain more detailed information.

[1219] Emotion recognition by emotion engine

[1220] The server uses generative AI and an emotion engine to analyze the user's emotions from video data in real time. The generative AI analyzes elements such as facial expressions, gestures, complexion, and gaze, and the emotion engine recognizes the user's emotional state based on that data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests.

[1221] Example: Facial expression analysis can quantify humorous reactions and levels of tension. For example, if a significant frown or lip twist is detected, that information is sent to an emotion engine, which then determines the emotional state, such as "tension" or "anxiety."

[1222] Creating communication methods

[1223] Based on the analysis results from the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods. Depending on the analysis results, it generates conversations that incorporate light-hearted jokes if the user is relaxed, or words that reassure the user if they are nervous.

[1224] Example: If a user is determined to be nervous, GPT-3 will generate advice such as "Take a deep breath and you'll feel better."

[1225] Example prompt sentence:

[1226] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[1227] Providing communication methods

[1228] The server sends the generated communication method to the device in real time and displays it on the user's UI via a REST API or notification service.

[1229] Example: The generated advice is displayed in a pop-up format in the upper right corner of the screen, making it easy for users to see.

[1230] User selection and execution

[1231] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that, enabling smooth communication that is close to face-to-face.

[1232] Example: A user selects the advice "Take a deep breath and you'll feel better" and verbally repeats it to the other person, making the conversation flow more naturally and effectively.

[1233] This system analyzes emotions in real time and provides effective communication methods, enabling high-quality dialogue even in remote environments. This process makes remote meetings and interviews as smooth as face-to-face meetings.

[1234] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1235] Step 1:

[1236] When a user starts a remote conference, the device activates the camera and microphone and acquires visual and audio data in real time. The device uses OpenCV to capture video data frame by frame (e.g., 30 frames per second) and stores audio data in a buffer. The acquired visual and audio data are input.

[1237] Specific operation: The camera captures the user's face, the microphone collects the user's speech, and the data is temporarily stored in the device's memory.

[1238] Input: User's visual and audio data

[1239] Output: Buffered encoded visual and audio data

[1240] Step 2:

[1241] The device transmits the visual and audio data stored in the buffer to the server at regular intervals (e.g., once per second) using a low-latency protocol such as WebSocket or RTMP.

[1242] Specific operation: The device opens a WebSocket and sends video frames and audio samples as binary data to the server.

[1243] Input: Buffered encoded visual and audio data

[1244] Output: Visual and audio data sent to the server

[1245] Step 3:

[1246] The server analyzes the visual and audio data received from the device. Image processing libraries such as OpenCV and Dlib, and deep learning frameworks such as TensorFlow and PyTorch are used for server-side processing. The received data is first decoded and then analyzed to extract visual and audio features.

[1247] Specific operation: The server decodes and decomposes the received data into frames, and uses OpenCV to extract features such as face and eye direction. At the same time, it also analyzes the features of the audio data.

[1248] Input: Visual and audio data sent to the server

[1249] Output: Analyzed visual and audio feature data

[1250] Step 4:

[1251] The server uses generative AI and an emotion engine to recognize the user's emotions in real time from the analyzed visual and audio feature data. The emotion engine uses deep learning models such as ResNet and VGG16 in combination with SVM and random forests to determine emotional states such as tension, joy, and concentration.

[1252] Specific operation: The emotion engine uses a neural network to analyze visual data such as facial expressions, gestures, skin tone, and gaze, and then combines this data with a machine learning algorithm to determine emotions.

[1253] Input: Analyzed visual and audio feature data

[1254] Output: Determined user's emotional state

[1255] Step 5:

[1256] Based on the analysis results of the emotion engine, the server uses natural language generation models such as BERT and GPT-3 to generate effective communication methods, such as light-hearted banter when the user is relaxed, or reassuring words when the user is nervous.

[1257] Specific operation: A generative AI model such as GPT-3 generates appropriate dialogue content using prompt sentences based on the results of sentiment analysis.

[1258] Example prompt sentence:

[1259] Generate appropriate ways to communicate if the user is nervous. Include simple suggestions or advice to help the user relax.

[1260] Input: Determined emotional state of the user

[1261] Output: The generated means of communication

[1262] Step 6:

[1263] The server sends the generated communication methods to the device in real time and displays them on the user's UI. This transmission uses a REST API or notification service.

[1264] Specific operation: The server converts the generated communication method into JSON format, sends it to the terminal via an HTTP request, and presents it on the terminal in a pop-up or dialog format.

[1265] Input: Generated communication method

[1266] Output: Communication options displayed on the device

[1267] Step 7:

[1268] Users can choose the appropriate communication method from the presented options and proceed with the conversation based on that choice, achieving smooth communication that is close to face-to-face.

[1269] Specific behavior: The user clicks or taps on an option presented in the UI and continues the conversation with the other person using that method.

[1270] Input: Communication options displayed on the device

[1271] Output: Dialogue content based on the communication method selected by the user

[1272] (Application example 2)

[1273] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1274] In traditional remote classes, it was difficult for teachers to instantly grasp students' emotional states and provide effective feedback at the appropriate time. Furthermore, the inability to provide appropriate explanations tailored to students' situations could result in a decline in educational effectiveness. This resulted in the quality of remote classes being inferior to face-to-face classes.

[1275] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1276] In this invention, the server includes: [means for transmitting acquired video data to the server;] [means for the server to analyze the video data;] [means for generating a communication method based on the analysis results;] [means for using smart glasses in an educational setting to present the generated communication method to a teacher in real time as feedback according to the student's emotional state; and [means for generating prompt sentences according to the emotional state and providing effective feedback.] This enables teachers to grasp students' emotional states in real time and provide appropriate feedback and supplementary explanations.

[1277] "Video data" refers to information in video frames captured using imaging equipment such as a camera.

[1278] A "server" is a computer system that processes and analyzes data over a network.

[1279] "Analysis" is the act of breaking down the information from the acquired data and examining its contents in detail.

[1280] "Generation" is the act of creating new results or means based on data or information.

[1281] A "terminal" is a hardware device that is directly operated by a user.

[1282] A "communication method" is a means or approach for effective dialogue or communication.

[1283] "Smart glasses" are glasses-type devices equipped with a camera and a display that can provide visual information.

[1284] "Feedback" is the act of returning information for the purpose of improvement or correction based on the information or results obtained.

[1285] A "prompt sentence" is an input sentence that is used to provide guidance and help to the generative AI model.

[1286] The "educational setting" refers to the place or environment where lessons and learning activities take place.

[1287] A "user" is a person or entity that uses a system or device.

[1288] "Emotional state" refers to the user's psychological state and fluctuations in feelings.

[1289] A system for specifically implementing this invention uses smart glasses, a camera, a server, a generative AI model, and an emotion engine, and the following embodiment is described in detail below.

[1290] First, the smart glasses are equipped with a camera that captures students' facial expressions, gestures, and gaze in real time during remote classes. The captured video data is temporarily stored in a buffer inside the smart glasses and then periodically sent to a server.

[1291] The server analyzes the received video data. The analysis includes a generative AI model and an emotion engine that recognizes the student's emotional state in real time based on facial expressions, gestures, complexion, gaze, and other factors. The generative AI model and emotion engine can accurately determine whether the student is in an emotional state such as happiness, confusion, or concentration.

[1292] Next, based on the analysis results, the server generates multiple effective communication methods. These methods are presented in real time to the teacher wearing the smart glasses. For example, if a student is confused, the teacher will provide feedback such as "Please explain again using easy-to-understand examples and metaphors."

[1293] Teachers can refer to these provided feedback and communication methods as they conduct their lessons, enabling them to provide feedback at the appropriate time based on students' level of understanding and emotional state, thereby improving the effectiveness of remote lessons.

[1294] Hardware and software used

[1295] Hardware: Smart glasses (with camera)

[1296] Software: OpenCV, EmotionRecognition library, CommunicationSuggestions module

[1297] Specific examples

[1298] For example, if the emotion engine recognizes that a student is confused during a lesson, the server will provide the teacher with feedback such as, "If the student is confused, please explain again using easy-to-understand examples and metaphors." The teacher can then use this feedback to re-explain the problem using concrete examples, such as, "This math problem might be easier to understand if you explain it using an apple."

[1299] Prompt Sentence Examples

[1300] "If students are confused, explain it again with an alternative, clearer example."

[1301] "If students are focused, keep them on pace."

[1302] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1303] Step 1:

[1304] The camera in the smart glasses captures video data of the students.

[1305] (Input) Real-time video of the student.

[1306] (Data processing) The camera sensor captures video frames and temporarily stores them in a buffer.

[1307] (Output) Video data for each frame.

[1308] Step 2:

[1309] The video data acquired by the smart glasses is sent to the server.

[1310] (Input) Video data stored in the buffer.

[1311] (Data processing) Video data is sent to the server reliably using network protocols.

[1312] (Output) Video data received by the server.

[1313] Step 3:

[1314] The server analyzes the received video data using a generative AI model and emotion engine.

[1315] (Input) Video data received by the server.

[1316] (Data processing) The generative AI model analyzes facial expressions, gestures, complexion, gaze, etc. for each frame, and the emotion engine uses this information to infer the student's emotional state.

[1317] (Output) Inferred emotional state (e.g., happy, confused, focused, etc.).

[1318] Step 4:

[1319] The server generates a plurality of effective communication methods based on the inferred emotional state.

[1320] (Input) Inferred emotional state.

[1321] (Data processing) Generate prompt sentences that correspond to the emotional state, and generate effective feedback methods using a generative AI model.

[1322] (Output) Multiple communication methods generated.

[1323] Step 5:

[1324] The server presents the generated communication method to the smart glasses in real time.

[1325] (Input) Multiple communication methods generated.

[1326] (Data processing) Displayed as text data on the smart glasses display.

[1327] (Output) Communication methods displayed on the smart glasses display.

[1328] Step 6:

[1329] The user, the teacher, selects the communication method presented and applies it to their lessons.

[1330] (Input) Communication methods displayed on the smart glasses display.

[1331] (Data processing) Teachers can choose to provide feedback at appropriate times according to students' emotional state.

[1332] (Output) Appropriate feedback from the teacher to the student.

[1333] Examples:

[1334] Example prompt sentence:

[1335] "If students are confused, explain it again with an alternative, clearer example."

[1336] "If students are focused, keep them on pace."

[1337] In this way, the system operates while clearly defining the specific input, data processing, and output at each step, allowing teachers to provide appropriate feedback to students in real time during remote classes based on their emotional state.

[1338] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1339] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1340] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1341] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1342] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1343] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1344] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1345] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1346] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1347] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1348] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1349] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1350] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1351] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1352] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1353] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1354] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1355] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1356] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1357] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1358] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1359] The following is further disclosed regarding the above embodiment.

[1360] (Claim 1)

[1361] [Means for acquiring video data;

[1362] [Means for transmitting the acquired video data to a server;

[1363] [Means for the server to analyze the video data;

[1364] [Means for generating a communication method based on the analysis results;

[1365] [Means for presenting the generated communication method to a terminal;

[1366] [a means for the user to select the communication methods presented; and

[1367] A system including:

[1368] (Claim 2)

[1369] [The system of claim 1, wherein the analysis infers emotions based on facial expressions, gestures, complexion, and gaze.

[1370] (Claim 3)

[1371] [The system according to claim 1, which presents the generated communication methods in multiple patterns.

[1372] "Example 1"

[1373] (Claim 1)

[1374] [Means for acquiring video data;

[1375] [Means for transmitting the acquired video data to a data processing device;

[1376] [Means for analyzing video data by a data processing device;

[1377] [Means for generating a communication method based on the analysis results;

[1378] [means for displaying the generated communication method on a display device;

[1379] [Means for users to select the communication methods presented; and

[1380] A system including:

[1381] (Claim 2)

[1382] [The system of claim 1, wherein the analysis infers emotions based on facial expressions, gestures, complexion, and gaze.

[1383] (Claim 3)

[1384] [The system according to claim 1, which presents the generated communication methods in multiple patterns.

[1385] "Application Example 1"

[1386] (Claim 1)

[1387] [Means for acquiring video data;

[1388] [Means for transmitting the acquired video data to a server;

[1389] [Means for the server to analyze the video data;

[1390] [Means for generating a communication method based on the analysis results;

[1391] [Means for presenting the generated communication method to a terminal;

[1392] [a means for the user to select the communication methods presented; and

[1393] [Means for generating and presenting real-time feedback based on the analysis results;

[1394] [A system that includes a means of providing appropriate instructions according to the worker's situation.]

[1395] (Claim 2)

[1396] [The system of claim 1, wherein the analysis infers emotions based on facial expressions, gestures, complexion, and gaze.

[1397] (Claim 3)

[1398] [The system according to claim 1, which presents the generated communication methods in multiple patterns.

[1399] "Example 2: Combining Emotion Engines"

[1400] (Claim 1)

[1401] [means for obtaining visual and audio data from a user;

[1402] [means for transmitting the acquired visual and audio data to a server;

[1403] [Means for the server to analyze the visual and audio data;

[1404] [Means for generating appropriate communication methods based on the analysis results;

[1405] [Means for presenting the generated communication means on a terminal;

[1406] [a means for the user to select the communication means presented; and

[1407] A system including:

[1408] (Claim 2)

[1409] [The system of claim 1, wherein the analysis infers emotions based on facial expressions, movements, skin tone, and gaze.

[1410] (Claim 3)

[1411] [The system according to claim 1, which presents the generated communication means as a plurality of options.

[1412] "Application example 2 when combining emotion engines"

[1413] (Claim 1)

[1414] [Means for acquiring video data;

[1415] [Means for transmitting the acquired video data to a server;

[1416] [Means for the server to analyze the video data;

[1417] [Means for generating a communication method based on the analysis results;

[1418] [Means for presenting the generated communication method to a terminal;

[1419] [a means for the user to select the communication methods presented; and

[1420] [A method for using smart glasses in an educational setting to present the generated communication methods to teachers in real time as feedback according to the students' emotional states; and

[1421] [Means for generating prompt sentences according to emotional states and providing effective feedback;

[1422] A system including:

[1423] (Claim 2)

[1424] [The system of claim 1, wherein the analysis infers emotions based on facial expressions, gestures, complexion, and gaze.

[1425] (Claim 3)

[1426] [The system according to claim 1, which presents the generated communication methods in multiple patterns. [Explanation of symbols]

[1427] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring video data; means for transmitting the acquired video data to a server; A means for the server to analyze the video data; means for generating a communication method based on the analysis results; means for presenting the generated communication method to a terminal; a means for the user to select the communication methods presented; A system including:

2. 2. The system of claim 1, wherein the emotion is inferred based on facial expressions, gestures, complexion, and gaze during the analysis.

3. The system according to claim 1, wherein the generated communication method is presented in a plurality of patterns.

Images