system

Abstract

Provide a system. [Solution] A means for registering participant information at the start of a conference; a means for capturing image data for each participant; means for analyzing emotions from the captured image data; A means for storing the analyzed emotion data in a database; a means for generating real-time feedback based on the emotion data; and means for notifying participants of the generated feedback.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the recent increase in teleworking, communication in remote environments often does not proceed smoothly. Furthermore, since it is difficult to convey emotional nuances, it is difficult to grasp the progress of business negotiations and provide feedback at the appropriate time. Furthermore, it is difficult to detect and prevent harassment in remote meetings. To improve this situation, a system that can analyze emotions in real time and provide appropriate feedback is needed. [Means for solving the problem]

[0005] This invention provides a system that registers participant information at the start of a meeting and captures image data of each participant. It includes a means for analyzing emotions from the captured image data and storing the analyzed emotional data in a database. It also provides a system that includes a means for generating feedback in real time based on the emotional data and notifying participants of the generated feedback. If the analyzed emotional data indicates doubt or anxiety, it has a function for suggesting countermeasures to the user. It also includes a means for analyzing the content of participants' comments and detecting whether the comments constitute harassment, and if harassment is detected, it issues a warning and alert. It also provides a function for continuously evaluating emotional data based on the progress of business negotiations, estimating the probability of success of the negotiations in real time, and notifying the user. These means facilitate communication in remote environments, preventing harassment, and managing the progress of business negotiations.

[0006] "Participant information" is identification information about each user participating in a conference, and includes user ID, name, job title, and the like.

[0007] "Image data" refers to still images or video frames of participants' faces captured via a camera.

[0008] "Emotion analysis" is the process of extracting and analyzing participants' emotional states (e.g., joy, sadness, anger, surprise, fear, calm, etc.) from image and audio data.

[0009] A "database" is a system that systematically stores, manages, and searches data such as emotional data and participant information.

[0010] "Feedback" is information that includes advice and instructions for the user, generated based on analyzed emotion data.

[0011] A "notification" is a means of conveying information to a user in real time, and includes on-screen pop-ups and audio alerts.

[0012] "Countermeasures" are content that instructs users on the actions and countermeasures they should take in response to negative emotions such as doubts and anxiety.

[0013] "Harassment detection" is the process of identifying inappropriate behavior or comments in real time based on the content of participants' comments.

[0014] "Negotiation progress" is a state that tracks changes in participants' reactions and emotions as the negotiation progresses.

[0015] "Probability of success" is an assessment of the likelihood that the objectives of a business negotiation or meeting will be achieved, expressed as a number such as a percentage. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention was developed to facilitate communication in remote environments. This system can register participant information at the start of a conference, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. Furthermore, it has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback.

[0038] The main process of the system is as follows:

[0039] 1. Starting a meeting

[0040] (User) The user launches the conferencing application and starts the conference.

[0041] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0042] 2. Capture image data

[0043] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0044] (Terminal) Sends the captured image data to the server in real time.

[0045] 3. Emotion analysis

[0046] (Server) The server sends the captured image data to an emotion analysis API (e.g., a general facial recognition API).

[0047] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[0048] (Server) Save the analysis results in the database.

[0049] 4. Generate feedback

[0050] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0051] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0052] 5. Feedback Notification

[0053] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0054] (User) The user selects and executes the appropriate action from the presented measures.

[0055] 6. Managing the progress of business negotiations

[0056] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0057] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0058] 7. Harassment Detection

[0059] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0060] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0061] Through these processes, the system of the present invention not only facilitates communication in remote environments and improves the success rate of business negotiations, but also prevents harassment from occurring. As a specific example, a situation can be imagined in which, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which will allow the business negotiation to proceed smoothly. This feedback function enables efficient communication in a remote environment, just like face-to-face communication.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] (User) The user launches the conferencing application and starts the conference.

[0065] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0066] Step 2:

[0067] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0068] (Terminal) Sends the captured image data to the server in real time.

[0069] Step 3:

[0070] (Server) The server receives the captured image data and sends it to the emotion analysis API.

[0071] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[0072] (Server) Save the analysis results in the database.

[0073] Step 4:

[0074] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0075] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0076] Step 5:

[0077] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0078] (User) The user selects and executes the appropriate action from the presented measures.

[0079] Step 6:

[0080] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0081] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0082] Step 7:

[0083] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0084] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] In today's remote meetings, physical distance often makes communication difficult, making it particularly difficult to recognize emotions and questions. It's also difficult to respond immediately when inappropriate remarks (harassment) are made, which reduces the productivity of meetings. To solve these issues, a system is needed that can analyze participants' emotions in real time and provide appropriate feedback.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for registering participant information at the start of a conference, means for acquiring image data of each participant, means for analyzing emotions from the acquired image data, means for saving the analyzed emotional data, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for analyzing participants' speech content and detecting inappropriate remarks. This enables recognition of emotions and provision of feedback in real time in remote conferences, thereby enabling smoother communication and improved conference productivity.

[0090] "Participant information" refers to data used to identify and specify individuals, such as user ID, name, job title, and email address.

[0091] "Image data" refers to digital images of participants' faces and expressions.

[0092] "Analyzing emotions" refers to the process of determining an emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.) from the acquired image data.

[0093] A "database" is an information system for storing and managing analyzed emotion data.

[0094] "Feedback" refers to notifications (e.g., specific actions or countermeasures) that are generated for participants based on the analyzed emotional data.

[0095] "Notify" means communicating the generated feedback to participants in the form of an on-screen pop-up, chat message, audio alert, etc.

[0096] "Utterance content" is the set of words and sentences spoken by participants during a conference.

[0097] "Inappropriate comments" refers to harassment or comments that are ethically questionable.

[0098] "Warnings and Alerts" are warning messages or notifications that are displayed when inappropriate remarks are detected.

[0099] The system of this invention aims to facilitate communication in a remote environment. The system has the functions of registering participant information at the start of a conference, capturing image data of each participant, analyzing emotions from the captured image data, and saving the analysis results in a database. Furthermore, the system provides a function of generating feedback in real time based on the emotion data and notifying the participants of the generated feedback. It also has a function of analyzing the content of participants' speech and detecting inappropriate remarks.

[0100] The program for realizing this system uses the following hardware and software.

[0101] Hardware used

[0102] Server: Responsible for conference session generation, database management, sentiment analysis and feedback generation.

[0103] Device: A computer or smartphone used by a participant, equipped with a camera and microphone to capture image and audio data.

[0104] Database: Stores the analyzed emotion data and information about the meeting.

[0105] Software used

[0106] Emotion analysis API: An API for analyzing emotions from image data (e.g., Microsoft® Azure® Face API or Google® Cloud Vision).

[0107] Natural language processing APIs: APIs for analyzing speech content and detecting inappropriate language (e.g., Amazon Comprehend or IBM Watson® Natural Language Understanding).

[0108] Processing flow

[0109] The main process of the system is as follows:

[0110] 1. Starting a meeting

[0111] A user launches a conferencing application and starts a conference.

[0112] The server generates a conference session, receives information about each participant, and registers it in a database.

[0113] 2. Capture and send image data

[0114] The device activates the camera and captures participants' faces in real time.

[0115] The image data captured by the terminal is periodically compressed and sent to the server.

[0116] 3. Sentiment analysis and data storage

[0117] The image data received by the server is sent to the emotion analysis API.

[0118] The emotion analysis API analyzes the image data, determines the emotional state, and returns the results to the server.

[0119] The server associates the analysis results with the corresponding user ID and stores them in a database.

[0120] 4. Feedback Generation and Notification

[0121] The server analyzes the emotion data stored in the database and generates feedback when a specific emotion is detected.

[0122] The device notifies the user of the generated feedback, such as via an on-screen pop-up or an audio alert.

[0123] 5. Analysis of speech content and warnings

[0124] The server sends the conversation content to a natural language processing API and analyzes it for inappropriate remarks.

[0125] If inappropriate remarks are detected, the server generates warnings and alerts and notifies the device.

[0126] Specific examples

[0127] For example, during a business meeting, if a user receives feedback such as "YYY has a question," the user can take action by "accepting questions," which will allow the business meeting to proceed smoothly. This allows for efficient communication in a remote environment, just like face-to-face communication.

[0128] Prompt Sentence Examples

[0129] Develop a system that analyzes participants' emotions during the negotiation and provides real-time feedback. For example, if a participant's image data indicates doubts or concerns, include a function that notifies the participant of this as feedback.

[0130] With the above specific examples and processing methods, this system can facilitate communication in remote environments, improve the success rate of business negotiations, and prevent harassment from occurring.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1: Start the meeting

[0133] Enter: The user launches the conferencing application and enters their authentication information.

[0134] Action: The user clicks the "Start Meeting" button.

[0135] Server process: The server creates a conference session, receives information about each participant, and registers it in a database. The participant information includes the user ID, name, job title, and email address.

[0136] Output: The new conference session ID and participant information are saved in the database.

[0137] Step 2: Capture image data

[0138] Input: Conference session ID and each participant's device to activate the camera.

[0139] How it works: The device activates the camera and captures participants' faces in real time.

[0140] Terminal processing: The captured image data is compressed into JPEG format and sent to the server at regular intervals (e.g., every second).

[0141] Output: The compressed image data is sent to the server.

[0142] Step 3: Sentiment Analysis

[0143] Input: Image data sent from the device.

[0144] Server processing: The server sends the received image data to an emotion analysis API (e.g., Microsoft Azure Face API).

[0145] How it works: The emotion analysis API processes image data to determine the participant's emotional state (e.g., happy, sad, anger, surprise, fear, neutral, etc.).

[0146] Output: The sentiment analysis results are returned to the server.

[0147] Step 4: Storing emotion data

[0148] Input: Analysis results returned from the sentiment analysis API.

[0149] Server processing: The server associates the analysis results with the corresponding user ID and saves them in the database.

[0150] How it works: The server stores emotion data in a database.

[0151] Output: Emotion data is recorded in a database.

[0152] Step 5: Generate feedback

[0153] Input: Emotion data stored in a database.

[0154] Server processing: The server analyzes the emotion data and generates feedback if certain emotions (e.g., doubt, anxiety) are detected.

[0155] Action: The server generates feedback such as "XXX seems to be having difficulty understanding. Please consider the following actions: 1. Accept questions 2. Provide additional explanation 3. Answer questions."

[0156] Output: The generated feedback is sent to the device.

[0157] Step 6: Notification of feedback

[0158] Input: Feedback sent by the server.

[0159] Device action: The device receives the feedback and notifies the user with an on-screen popup, chat message, audio alert, etc.

[0160] Action: The device displays or sounds the feedback to the user.

[0161] Output: The user receives feedback.

[0162] Step 7: Analyzing the speech

[0163] Input: What participants say during the meeting.

[0164] Server processing: The server sends the utterance content to a natural language processing API (e.g., Amazon Comprehend).

[0165] How it works: A natural language processing API analyzes speech and detects inappropriate language.

[0166] Output: If inappropriate comments are detected, the analysis results are returned to the server.

[0167] Step 8: Notification of warnings and alerts

[0168] Input: Analysis results returned from the natural language processing API.

[0169] Server Processing: The server detects inappropriate comments and generates warnings and alerts.

[0170] What it does: The server generates a warning such as "That comment is inappropriate" and sends an alert to an administrator.

[0171] Output: Warnings and alerts are sent to the terminal and to the administrator.

[0172] These processing steps enable real-time emotion analysis and feedback during remote meetings, facilitating smoother communication and improving meeting productivity.

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] In remote communication, it is more difficult to grasp nonverbal emotions than in face-to-face conversations, which can hinder the progress of business negotiations, customer service, and other communication. It is also difficult to properly understand and quickly respond to feelings of anxiety and doubt during conversations with customers. Furthermore, harassment is on the rise in remote environments, making detection and response to it a critical issue.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for saving the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for capturing customer facial image data, means for analyzing customer emotions from the captured facial image data, means for saving the customer emotional data in a database, and means for notifying staff in real time of the customer's response based on the emotional data. This makes it possible to grasp the emotional state of customers even in a remote environment, and to provide feedback to increase the success rate of business negotiations and detect harassment.

[0178] "At the start of the meeting" refers to the moment when a remote meeting or business negotiation officially begins, and refers to the state in which the preparation stage is completed before that.

[0179] "Participant information" refers to details about each person attending a meeting or business meeting, including, for example, name, job title, user ID, etc.

[0180] "Image data" refers to still images and videos captured by an imaging device such as a camera.

[0181] "Emotion analysis" refers to the process of determining a person's emotional state from acquired image data using specific algorithms or APIs.

[0182] A "database" refers to a system or software that stores information in an organized manner and makes it easy to search and update.

[0183] "Feedback" refers to advice or information generated based on the analyzed emotional data and provided as a response to the user.

[0184] "Real-time" refers to a state in which processing and reaction occur almost instantaneously, meaning that there is very little time delay.

[0185] "Notifying participants" refers to the act of informing people participating in a meeting or business negotiation of information.

[0186] "Customer" means a person who visits the Virtual Store to purchase or use goods or services.

[0187] "Capturing" refers to obtaining image data using an imaging device such as a camera.

[0188] "Staff" refers to the person responsible for dealing with customers in the virtual store.

[0189] This invention is a system for facilitating communication primarily in remote environments, and provides the function of registering participant information at the start of a meeting or business negotiation, analyzing the emotions of each participant and customer in real time, and notifying them of the feedback. Specific programs and processing methods for realizing this invention are described below.

[0190] First, at the start of a meeting or business negotiation, the user launches an application, and the server generates a session for the meeting or business negotiation. Basic information about participants and customers is registered, and preparations for the meeting or business negotiation are completed.

[0191] Next, a camera built into the device (e.g., smart glasses or a head-mounted display) captures the faces of participants or customers in real time, and the captured image data is immediately sent to a cloud server.

[0192] Next, the cloud server receives the transmitted image data and analyzes it using an emotion analysis API (e.g., Google Cloud Vision API or Microsoft Azure Face API). At this time, the emotional state (e.g., joy, anxiety, anger, etc.) is determined and the results are stored in a database.

[0193] Based on the obtained emotional data, the server generates feedback in real time. For example, if the customer looks anxious, the server generates feedback such as "The customer is feeling anxious. He or she may need help." This feedback is immediately displayed on the device screen (e.g., smart glasses), allowing the user to take appropriate action.

[0194] Additionally, the server continuously evaluates sentiment data and generates alerts on the progress of sales negotiations and the probability of success of customer interactions, allowing additional measures to be taken at the appropriate time.

[0195] For example, if a customer looks anxious while browsing products in a virtual store, the smart glasses will display a notification saying, "The customer is anxious and may need help." This allows staff to respond quickly and provide efficient customer service.

[0196] The system also uses a natural language processing API to analyze participants' comments and constantly monitor for signs of harassment. If a comment is deemed inappropriate, it immediately displays a warning and sends an alert to an administrator.

[0197] An example of a prompt to input to a generative AI model is:

[0198] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[0199] In this way, the system integrates a variety of technologies to improve the quality of interactions and increase customer satisfaction in remote environments.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1: Start the meeting

[0202] A user launches an application and starts a meeting or business meeting. The server receives this information and creates a meeting or business meeting session. The input includes the user ID, meeting ID, and participant information, and the output is the start of a meeting session.

[0203] Step 2: Register participant information

[0204] The server registers information about each participant, including name, job title, user ID, etc. The input includes basic participant information, and the output generates participant information registered in the database.

[0205] Step 3: Capture image data

[0206] The device (such as smart glasses) captures the face of a participant or customer in real time using a built-in camera and generates image data. The input is a face image, and the output is the captured image data.

[0207] Step 4: Sending image data

[0208] The device sends the captured image data to the cloud server. The input is the captured image data, and the output is the image data sent to the server.

[0209] Step 5: Sentiment Analysis

[0210] The server sends the received image data to an emotion analysis API and obtains the analysis results. The input includes image data, and the emotional state is output as a result of facial recognition and emotion analysis. Specifically, Google Cloud Vision API and Microsoft Azure Face API are used.

[0211] Step 6: Storing Emotion Data

[0212] The server stores the analysis results in a database. The input is the emotion analysis results, and the output is the emotion data stored in the database.

[0213] Step 7: Generate feedback

[0214] The server generates real-time feedback based on the emotion data stored in the database. The input includes the analyzed emotion data, and the output is the generated feedback. For example, if the customer is feeling anxious, the generated feedback is "The customer is feeling anxious. He or she may need help."

[0215] Step 8: Notification of feedback

[0216] The terminal notifies the user of the generated feedback. The input is the generated feedback, and the output is a notification to the user. This notification is displayed, for example, as a pop-up on the display of the smart glasses.

[0217] Step 9: Analyzing Speech

[0218] The server analyzes participants' comments in real time to detect signs of harassment. The input includes audio data, and a natural language processing API is used to determine whether the comments are appropriate, and the analysis results are output.

[0219] Step 10: Harassment Warning

[0220] If the server detects signs of harassment, it immediately displays a warning and sends an alert to the administrator. The input is the analyzed speech data, and the output is a warning notification to the user and an alert to the administrator.

[0221] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[0222] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[0223] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0224] The system of the present invention was developed to facilitate smooth communication in remote environments. This system can register participant information at the start of a meeting, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[0225] The main process of the system is as follows:

[0226] 1. Starting a meeting

[0227] (User) The user launches the conferencing application and starts the conference.

[0228] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0229] 2. Capture image data

[0230] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0231] (Terminal) Image and audio data captured in real time is sent to the server.

[0232] 3. Emotion analysis

[0233] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[0234] (Server) The emotion analysis API and emotion engine process the image and audio data, determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and return the results to the server. The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[0235] (Server) Save the analysis results in the database.

[0236] 4. Generate feedback

[0237] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0238] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0239] 5. Feedback Notification

[0240] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0241] (User) The user selects and executes the appropriate action from the presented measures.

[0242] 6. Managing the progress of business negotiations

[0243] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0244] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0245] 7. Harassment Detection

[0246] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0247] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0248] This system can facilitate smooth communication in a remote environment, improve the success rate of business negotiations, and prevent harassment from occurring. For example, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which allows the business negotiation to proceed smoothly. This makes it possible to achieve effective communication in a remote environment, just like face-to-face communication.

[0249] The processing flow will be explained below.

[0250] Step 1:

[0251] (User) The user launches the conferencing application and starts the conference.

[0252] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0253] Step 2:

[0254] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0255] (Terminal) Image and audio data captured in real time is sent to the server.

[0256] Step 3:

[0257] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[0258] (Server) The emotion analysis API and emotion engine process the image and audio data to determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.). The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[0259] (Server) Save the analysis results in the database.

[0260] Step 4:

[0261] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0262] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0263] Step 5:

[0264] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0265] (User) The user selects and executes the appropriate action from the presented measures.

[0266] Step 6:

[0267] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0268] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0269] Step 7:

[0270] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0271] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0272] Example 2

[0273] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] In remote meetings, there is a need to grasp participants' emotional states in real time, ensuring smooth business negotiations while preventing harassment. However, conventional systems lack sufficient emotional analysis and feedback generation, which can significantly impact the success of business negotiations. In particular, if appropriate measures are not provided when participants have doubts or concerns, communication efficiency decreases, affecting the success of business negotiations. Furthermore, if harassment is not detected and addressed early, the safety and security of participants in remote meetings may be threatened. To solve these issues, a system is needed that can accurately analyze participants' emotional states and quickly provide appropriate feedback.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0276] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image and audio data of each participant, means for analyzing emotions from the captured image and audio data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for continuously evaluating the progress of the business negotiation and grasping the situation, means for analyzing the content of participants' comments and detecting whether the comments constitute harassment, and means for issuing warnings and alerts if harassment is detected. This allows for a real-time understanding of participants' emotional states in remote conferences and the provision of appropriate feedback and countermeasures, thereby facilitating smooth communication and improving the success of business negotiations. Furthermore, preventing harassment before it occurs ensures participants' peace of mind and safety.

[0277] "Participant information" refers to personal identification information such as user ID, name, and job title registered in the conference system.

[0278] "Image data" is digital data of still or moving images captured by a camera.

[0279] "Audio data" is digital data of audio captured by a microphone.

[0280] "Emotion analysis" is the process of determining the emotional state of participants based on image and audio data using specific algorithms or engines.

[0281] A "database" is a system for efficiently storing, retrieving, and managing large amounts of digital data.

[0282] "Feedback" refers to notifications or instructions to participants that are generated based on analyzed emotional data.

[0283] "Deal progress" refers to indicators or data for assessing whether a deal is progressing smoothly or heading towards success.

[0284] "Harassment detection" is the process of analyzing participants' comments and identifying whether they constitute inappropriate or harassing content.

[0285] A "warning" is a warning message that is displayed to participants when harassment is detected.

[0286] An "alert" is a warning that is sent to a system administrator when harassment is detected.

[0287] The system of the present invention was developed to facilitate smooth communication in remote environments. This system registers participant information at the start of a conference, captures image data of each participant, analyzes emotions from the captured image and audio data, and stores the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[0288] A specific embodiment for implementing the system is as follows.

[0289] Hardware and software used

[0290] 1. User device: A computer or smart device (e.g., PC, tablet, smartphone) with a camera and microphone.

[0291] 2. Server: An integrated management system equipped with a high-performance processor that performs emotion analysis and database operations.

[0292] 3. Conferencing application: Software that allows users to launch and manage meetings (e.g., Zoom, MICROSOFT TEAMS®).

[0293] 4. Sentiment Analysis API: Cloud services for analyzing emotional states (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API).

[0294] Description of the main process

[0295] Starting a meeting

[0296] A user launches a conference application and starts a conference. The user clicks the "Start Conference" button and enters the conference details (title, date and time, participants, etc.).

[0297] The server receives the user's request, creates a conference session (e.g., a unique conference ID identified by a UUID), and saves the participant information from the user's input into a database.

[0298] Image data capture

[0299] The device activates the camera and captures the participant's face as a real-time video feed, capturing frames at regular intervals (e.g., 30 frames per second).

[0300] The device transmits video data captured in real time and audio data obtained from the microphone to the server in streaming format. Security is ensured by encrypting the transmitted data.

[0301] Emotion analysis

[0302] The server receives the video and audio data sent from the terminal and checks the consistency of the received data.

[0303] The server sends the received data to the sentiment analysis API and the sentiment engine, and performs preprocessing (e.g., face recognition and cropping) if necessary.

[0304] The emotion analysis API analyzes the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns the results to the server.

[0305] The server generates complex emotional data and stores the analysis results in a database.

[0306] Feedback generation and notification

[0307] The server queries the emotional data stored in the database and generates feedback if a specific emotional state is detected, e.g., "XXX is in doubt. You should take the following actions: 1. Answer the question 2. Provide further explanation 3. Accept the question."

[0308] The device receives the feedback and notifies the user, either via a pop-up on the screen or an audio alert.

[0309] The user selects and executes the appropriate action from the presented countermeasures.

[0310] Negotiation progress management

[0311] The server evaluates the emotional data in real time to understand the progress of the negotiation and calculates the balance between positive and negative emotions.

[0312] The server evaluates the chances of success of the deal and generates specific alerts depending on the specific situation, e.g. "Given your current emotional state, the chances of success of the deal are 70%."

[0313] Harassment Detection

[0314] The server analyzes participants' comments using a natural language processing API to identify whether they constitute inappropriate or harassing content.

[0315] If the server detects any inappropriate comments, it will immediately display a warning and send an alert to the administrator, for example, a warning message will be displayed on the device saying "That comment is inappropriate."

[0316] Examples of concrete examples and prompts

[0317] As an example, consider a situation where a user receives feedback during a business meeting saying, "YYY has a question." In this case, the user can take the measure of "accepting questions," allowing the business meeting to proceed smoothly. As a result, effective communication similar to face-to-face communication can be achieved even in a remote environment.

[0318] Prompt Sentence Examples

[0319] "Analyze participants' emotions during a meeting."

[0320] "Generate measures to take if person Y feels anxious."

[0321] "Calculate the probability of the meeting being successful."

[0322] In this way, this system facilitates communication in a remote environment, improves the chances of successful business negotiations, and prevents harassment from occurring.

[0323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0324] Step 1:

[0325] Starting a meeting

[0326] The user starts the conference application and clicks the "Start Conference" button. Detailed information about the conference (title, date and time, participant information) is required as input.

[0327] The server receives the user's input information and creates a conference session. By saving the conference ID and participant information in a database, the output is the establishment of a new conference session and the registration of participant information.

[0328] Step 2:

[0329] Image data capture

[0330] The device activates the camera as soon as the meeting starts and captures participants' faces in real time, and requires a video feed from the camera as input.

[0331] The device continuously captures video frames in real time (e.g., 30 frames per second) and also receives audio data from the microphone. The output is the captured video and audio data.

[0332] Step 3:

[0333] Transmission of image data and audio data

[0334] The device encrypts the captured video and audio data and sends them to the server in streaming format. The data captured by the device is required as input.

[0335] The server receives these data and checks their integrity. As an output, the data whose integrity has been confirmed is stored on the server.

[0336] Step 4:

[0337] Emotion analysis

[0338] The server sends the received video and audio data to an emotion analysis API (e.g., Microsoft Azure Emotion API) and emotion engine. The received data is used as input.

[0339] The emotion analysis API analyzes the video and audio data, calculates the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns it to the server. The analysis results (emotional state and confidence level) are obtained as output.

[0340] The server generates composite emotion data and stores it in a database.

[0341] Step 5:

[0342] Generate feedback

[0343] The server queries the emotion data stored in the database and generates feedback if a specific emotional state (e.g., doubt, anxiety) is detected in real time. As input, it requires emotion data in the database.

[0344] The server generates feedback (e.g., "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept the question") as output.

[0345] Step 6:

[0346] Feedback Notification

[0347] The terminal notifies the user of the feedback received from the server. Feedback data is required as input.

[0348] The device will notify the user of the feedback via a pop-up on the screen or an audio alert, and the notified feedback will be available as output.

[0349] Step 7:

[0350] Negotiation progress management

[0351] The server continuously evaluates the emotion data and grasps the progress of the business negotiation in real time. Continuous emotion data is required as input.

[0352] The server evaluates the probability of success of the negotiation and generates a specific alert (e.g., "Based on your current emotional state, the probability of success of the negotiation is 70%) according to the progress. The progress management alert is obtained as an output.

[0353] Step 8:

[0354] Harassment Detection

[0355] The server analyzes participants' speech in real time using a natural language processing API. Voice data is required as input.

[0356] If the server detects harassing comments, it will immediately generate a warning and alert (e.g., "That comment is inappropriate"). The output will be a warning and an alert to the administrator.

[0357] These steps will ensure that remote meetings and business negotiations proceed smoothly, enable participants to understand their emotional state, and provide appropriate feedback. They will also help prevent harassment and create an environment where participants can feel safe and secure.

[0358] (Application example 2)

[0359] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Many challenges remain in facilitating communication and improving the success rate of business negotiations in remote environments. In particular, in factory environments, it is difficult to grasp the emotional state of workers in real time, which impacts work efficiency and safety. Furthermore, early detection and countermeasures for harassment are insufficient, making it difficult to maintain a healthy work environment. A method is needed to overcome these challenges and improve worker safety and work efficiency.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0362] In this invention, the server includes: means for registering participant information at the start of a conference; means for capturing image data of each participant; means for analyzing emotions from the captured image data; means for storing the analyzed emotional data in a database; means for generating feedback in real time based on the emotional data; means for notifying participants of the generated feedback; means for analyzing the emotional state of factory workers in their work environment and providing real-time feedback; and means for detecting signs of harassment during work and sending an alert to a manager. This enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. Furthermore, it facilitates early detection of harassment and appropriate countermeasures.

[0363] A "meeting" is a place where multiple participants gather based on a specific purpose or agenda to share information, discuss, and make decisions.

[0364] "Participant information" is information that identifies and identifies people who participate in a meeting or work environment, such as name, job title, and user ID.

[0365] "Image data" refers to digital image files containing visual data captured by a camera or other device.

[0366] "Emotion analysis" is the process of analyzing the emotional state of participants or workers based on their facial expressions, voice, behavior, etc. from captured image data.

[0367] A "database" is a system for systematically storing and managing analyzed emotional data and other information.

[0368] "Feedback" refers to real-time information and advice provided to users based on emotional data.

[0369] A "factory worker" is a person who engages in production activities through mechanical or manual labor in a manufacturing environment.

[0370] The "work environment" refers to the place where workers perform their daily work, such as a factory or manufacturing site, and the surrounding environment.

[0371] "Real time" refers to a state in which processing and feedback are carried out simultaneously and immediately.

[0372] "Harassment" refers to inappropriate and offensive words, actions, or behavior in the work environment that have a negative impact on the health and work of workers.

[0373] A "warning" is a message that alerts users and administrators when inappropriate behavior such as harassment is detected.

[0374] A "system" is a comprehensive operational structure that integrates various means in a meeting or work environment.

[0375] This invention provides a system for analyzing the emotional states of workers in a factory environment in real time and providing efficient feedback. The system includes means for registering participant information at the start of a meeting, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for detecting signs of harassment during work and sending a warning to a manager.

[0376] The server registers participant information at the start of a meeting. This information includes user ID, name, and job title. This makes it possible to identify each participant. Each participant's device is equipped with a camera, which captures image data of the participant in real time. The captured image data is sent to the server, where emotion analysis is performed using EmotionAPI.

[0377] The results of emotion analysis are stored in a database. EmotionAPI works in conjunction with image processing libraries such as OpenCV to analyze facial expressions and vocal characteristics from captured image data. Feedback is generated in real time based on the results of this analysis. For example, if a worker feels doubtful or anxious, the server will generate feedback such as, "This worker has doubts. Please provide assistance."

[0378] The generated feedback is sent to the worker's device, and a similar notification is also sent to the manager. This allows the worker to receive appropriate support and improves work efficiency. The server also analyzes the content of comments made while working, and if inappropriate comments are detected, a warning is immediately displayed and an alert is sent to the manager. This enables early detection and countermeasures against harassment.

[0379] For example, when the system is started at the start of work, it automatically captures the worker's face and performs real-time emotional analysis using the Emotion API. If emotions such as "doubt" or "anxiety" are detected during work, the robot's display will display "This worker has doubts. Help is needed," and a notification will also be sent to the manager.

[0380] An example of a prompt statement can be written as follows:

[0381] Create a factory management system that combines image capture and emotion analysis. Include a function that analyzes the emotional state of workers, generates feedback when doubts or concerns are detected, and notifies workers and managers.

[0382] With this configuration, the present invention enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. It also facilitates early detection of harassment and the implementation of appropriate countermeasures.

[0383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0384] Step 1:

[0385] The server registers participant information when a conference starts. Specifically, a user starts the conference application and inputs participant information such as user ID, name, and job title. This data is sent to the server and stored in a database. This makes it possible to identify participants. The input data is user ID, name, and job title, and the output data is the participant information stored in the database.

[0386] Step 2:

[0387] The device activates the camera and captures image data of each participant. The camera photographs the participants' faces in real time and sends the image data to the server. The input data is the photographed image data, and the output data is the image data sent to the server.

[0388] Step 3:

[0389] The server sends the received image data to the Emotion API for emotion analysis. The Emotion API uses OpenCV to analyze facial expressions and other features from the image data. The analysis results are output as emotional states such as joy, sadness, anger, surprise, fear, and calm. The input data is the image data, and the output data is the analyzed emotional state.

[0390] Step 4:

[0391] The server stores the analyzed emotional data in a database. The emotional states obtained as a result of the analysis (e.g., joy, sadness, anger, surprise, fear, calm, etc.) are stored in association with the corresponding participant information in the database. The input data is the result of the emotion analysis, and the output data is the emotional data stored in the database.

[0392] Step 5:

[0393] The server generates real-time feedback based on the emotion data stored in the database. For example, if the emotion data contains doubts or anxiety, the server generates feedback such as "This worker is feeling doubtful. Please provide assistance." The input data is the stored emotion data, and the output data is the generated feedback.

[0394] Step 6:

[0395] The generated feedback is notified to the device. The device notifies the user of the feedback content by displaying a pop-up or by audio alert. The user checks the notified feedback and takes appropriate action. The input data is the generated feedback, and the output data is the feedback notification.

[0396] Step 7:

[0397] The server analyzes what is said during work and monitors for signs of harassment. It uses a natural language processing API to analyze the content of speech, and if inappropriate behavior is detected, it generates warnings and alerts and notifies administrators. The input data is what the worker says, and the output data is the warnings and alerts that are the results of the analysis.

[0398] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0399] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0400] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0401] [Second embodiment]

[0402] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0403] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0404] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0405] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0406] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0408] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0409] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0410] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0411] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0412] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0413] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0414] The system of the present invention was developed to facilitate communication in remote environments. This system can register participant information at the start of a conference, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. Furthermore, it has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback.

[0415] The main process of the system is as follows:

[0416] 1. Starting a meeting

[0417] (User) The user launches the conferencing application and starts the conference.

[0418] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0419] 2. Capture image data

[0420] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0421] (Terminal) Sends the captured image data to the server in real time.

[0422] 3. Emotion analysis

[0423] (Server) The server sends the captured image data to an emotion analysis API (e.g., a general facial recognition API).

[0424] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[0425] (Server) Save the analysis results in the database.

[0426] 4. Generate feedback

[0427] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0428] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0429] 5. Feedback Notification

[0430] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0431] (User) The user selects and executes the appropriate action from the presented measures.

[0432] 6. Managing the progress of business negotiations

[0433] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0434] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0435] 7. Harassment Detection

[0436] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0437] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0438] Through these processes, the system of the present invention not only facilitates communication in remote environments and improves the success rate of business negotiations, but also prevents harassment from occurring. As a specific example, a situation can be imagined in which, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which will allow the business negotiation to proceed smoothly. This feedback function enables efficient communication in a remote environment, just like face-to-face communication.

[0439] The processing flow will be explained below.

[0440] Step 1:

[0441] (User) The user launches the conferencing application and starts the conference.

[0442] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0443] Step 2:

[0444] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0445] (Terminal) Sends the captured image data to the server in real time.

[0446] Step 3:

[0447] (Server) The server receives the captured image data and sends it to the emotion analysis API.

[0448] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[0449] (Server) Save the analysis results in the database.

[0450] Step 4:

[0451] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0452] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0453] Step 5:

[0454] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0455] (User) The user selects and executes the appropriate action from the presented measures.

[0456] Step 6:

[0457] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0458] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0459] Step 7:

[0460] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0461] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0462] Example 1

[0463] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0464] In today's remote meetings, physical distance often makes communication difficult, making it particularly difficult to recognize emotions and questions. It's also difficult to respond immediately when inappropriate remarks (harassment) are made, which reduces the productivity of meetings. To solve these issues, a system is needed that can analyze participants' emotions in real time and provide appropriate feedback.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0466] In this invention, the server includes means for registering participant information at the start of a conference, means for acquiring image data of each participant, means for analyzing emotions from the acquired image data, means for saving the analyzed emotional data, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for analyzing participants' speech content and detecting inappropriate remarks. This enables recognition of emotions and provision of feedback in real time in remote conferences, thereby enabling smoother communication and improved conference productivity.

[0467] "Participant information" refers to data used to identify and specify individuals, such as user ID, name, job title, and email address.

[0468] "Image data" refers to digital images of participants' faces and expressions.

[0469] "Analyzing emotions" refers to the process of determining an emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.) from the acquired image data.

[0470] A "database" is an information system for storing and managing analyzed emotion data.

[0471] "Feedback" refers to notifications (e.g., specific actions or countermeasures) that are generated for participants based on the analyzed emotional data.

[0472] "Notify" means communicating the generated feedback to participants in the form of an on-screen pop-up, chat message, audio alert, etc.

[0473] "Utterance content" is the set of words and sentences spoken by participants during a conference.

[0474] "Inappropriate comments" refers to harassment or comments that are ethically questionable.

[0475] "Warnings and Alerts" are warning messages or notifications that are displayed when inappropriate remarks are detected.

[0476] The system of this invention aims to facilitate communication in a remote environment. The system has the functions of registering participant information at the start of a conference, capturing image data of each participant, analyzing emotions from the captured image data, and saving the analysis results in a database. Furthermore, the system provides a function of generating feedback in real time based on the emotion data and notifying the participants of the generated feedback. It also has a function of analyzing the content of participants' speech and detecting inappropriate remarks.

[0477] The program for realizing this system uses the following hardware and software.

[0478] Hardware used

[0479] Server: Responsible for conference session generation, database management, sentiment analysis and feedback generation.

[0480] Device: A computer or smartphone used by a participant, equipped with a camera and microphone to capture image and audio data.

[0481] Database: Stores the analyzed emotion data and information about the meeting.

[0482] Software used

[0483] Sentiment Analysis API: An API for analyzing emotions from image data (e.g., Microsoft Azure Face API or Google Cloud Vision).

[0484] Natural language processing APIs: APIs for analyzing speech and detecting inappropriate language (e.g., Amazon Comprehend or IBM Watson Natural Language Understanding).

[0485] Processing flow

[0486] The main process of the system is as follows:

[0487] 1. Starting a meeting

[0488] A user launches a conferencing application and starts a conference.

[0489] The server generates a conference session, receives information about each participant, and registers it in a database.

[0490] 2. Capture and send image data

[0491] The device activates the camera and captures participants' faces in real time.

[0492] The image data captured by the terminal is periodically compressed and sent to the server.

[0493] 3. Sentiment analysis and data storage

[0494] The image data received by the server is sent to the emotion analysis API.

[0495] The emotion analysis API analyzes the image data, determines the emotional state, and returns the results to the server.

[0496] The server associates the analysis results with the corresponding user ID and stores them in a database.

[0497] 4. Feedback Generation and Notification

[0498] The server analyzes the emotion data stored in the database and generates feedback when a specific emotion is detected.

[0499] The device notifies the user of the generated feedback, such as via an on-screen pop-up or an audio alert.

[0500] 5. Analysis of speech content and warnings

[0501] The server sends the conversation content to a natural language processing API and analyzes it for inappropriate remarks.

[0502] If inappropriate remarks are detected, the server generates warnings and alerts and notifies the device.

[0503] Specific examples

[0504] For example, during a business meeting, if a user receives feedback such as "YYY has a question," the user can take action by "accepting questions," which will allow the business meeting to proceed smoothly. This allows for efficient communication in a remote environment, just like face-to-face communication.

[0505] Prompt Sentence Examples

[0506] Develop a system that analyzes participants' emotions during the negotiation and provides real-time feedback. For example, if a participant's image data indicates doubts or concerns, include a function that notifies the participant of this as feedback.

[0507] With the above specific examples and processing methods, this system can facilitate communication in remote environments, improve the success rate of business negotiations, and prevent harassment from occurring.

[0508] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0509] Step 1: Start the meeting

[0510] Enter: The user launches the conferencing application and enters their authentication information.

[0511] Action: The user clicks the "Start Meeting" button.

[0512] Server process: The server creates a conference session, receives information about each participant, and registers it in a database. The participant information includes the user ID, name, job title, and email address.

[0513] Output: The new conference session ID and participant information are saved in the database.

[0514] Step 2: Capture image data

[0515] Input: Conference session ID and each participant's device to activate the camera.

[0516] How it works: The device activates the camera and captures participants' faces in real time.

[0517] Terminal processing: The captured image data is compressed into JPEG format and sent to the server at regular intervals (e.g., every second).

[0518] Output: The compressed image data is sent to the server.

[0519] Step 3: Sentiment Analysis

[0520] Input: Image data sent from the device.

[0521] Server processing: The server sends the received image data to an emotion analysis API (e.g., Microsoft Azure Face API).

[0522] How it works: The emotion analysis API processes image data to determine the participant's emotional state (e.g., happy, sad, anger, surprise, fear, neutral, etc.).

[0523] Output: The sentiment analysis results are returned to the server.

[0524] Step 4: Storing emotion data

[0525] Input: Analysis results returned from the sentiment analysis API.

[0526] Server processing: The server associates the analysis results with the corresponding user ID and saves them in the database.

[0527] How it works: The server stores emotion data in a database.

[0528] Output: Emotion data is recorded in a database.

[0529] Step 5: Generate feedback

[0530] Input: Emotion data stored in a database.

[0531] Server processing: The server analyzes the emotion data and generates feedback if certain emotions (e.g., doubt, anxiety) are detected.

[0532] Action: The server generates feedback such as "XXX seems to be having difficulty understanding. Please consider the following actions: 1. Accept questions 2. Provide additional explanation 3. Answer questions."

[0533] Output: The generated feedback is sent to the device.

[0534] Step 6: Notification of feedback

[0535] Input: Feedback sent by the server.

[0536] Device action: The device receives the feedback and notifies the user with an on-screen popup, chat message, audio alert, etc.

[0537] Action: The device displays or sounds the feedback to the user.

[0538] Output: The user receives feedback.

[0539] Step 7: Analyzing the speech

[0540] Input: What participants say during the meeting.

[0541] Server processing: The server sends the utterance content to a natural language processing API (e.g., Amazon Comprehend).

[0542] How it works: A natural language processing API analyzes speech and detects inappropriate language.

[0543] Output: If inappropriate comments are detected, the analysis results are returned to the server.

[0544] Step 8: Notification of warnings and alerts

[0545] Input: Analysis results returned from the natural language processing API.

[0546] Server Processing: The server detects inappropriate comments and generates warnings and alerts.

[0547] What it does: The server generates a warning such as "That comment is inappropriate" and sends an alert to an administrator.

[0548] Output: Warnings and alerts are sent to the terminal and to the administrator.

[0549] These processing steps enable real-time emotion analysis and feedback during remote meetings, facilitating smoother communication and improving meeting productivity.

[0550] (Application example 1)

[0551] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0552] In remote communication, it is more difficult to grasp nonverbal emotions than in face-to-face conversations, which can hinder the progress of business negotiations, customer service, and other communication. It is also difficult to properly understand and quickly respond to feelings of anxiety and doubt during conversations with customers. Furthermore, harassment is on the rise in remote environments, making detection and response to it a critical issue.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0554] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for saving the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for capturing customer facial image data, means for analyzing customer emotions from the captured facial image data, means for saving the customer emotional data in a database, and means for notifying staff in real time of the customer's response based on the emotional data. This makes it possible to grasp the emotional state of customers even in a remote environment, and to provide feedback to increase the success rate of business negotiations and detect harassment.

[0555] "At the start of the meeting" refers to the moment when a remote meeting or business negotiation officially begins, and refers to the state in which the preparation stage is completed before that.

[0556] "Participant information" refers to details about each person attending a meeting or business meeting, including, for example, name, job title, user ID, etc.

[0557] "Image data" refers to still images and videos captured by an imaging device such as a camera.

[0558] "Emotion analysis" refers to the process of determining a person's emotional state from acquired image data using specific algorithms or APIs.

[0559] A "database" refers to a system or software that stores information in an organized manner and makes it easy to search and update.

[0560] "Feedback" refers to advice or information generated based on the analyzed emotional data and provided as a response to the user.

[0561] "Real-time" refers to a state in which processing and reaction occur almost instantaneously, meaning that there is very little time delay.

[0562] "Notifying participants" refers to the act of informing people participating in a meeting or business negotiation of information.

[0563] "Customer" means a person who visits the Virtual Store to purchase or use goods or services.

[0564] "Capturing" refers to obtaining image data using an imaging device such as a camera.

[0565] "Staff" refers to the person responsible for dealing with customers in the virtual store.

[0566] This invention is a system for facilitating communication primarily in remote environments, and provides the function of registering participant information at the start of a meeting or business negotiation, analyzing the emotions of each participant and customer in real time, and notifying them of the feedback. Specific programs and processing methods for realizing this invention are described below.

[0567] First, at the start of a meeting or business negotiation, the user launches an application, and the server generates a session for the meeting or business negotiation. Basic information about participants and customers is registered, and preparations for the meeting or business negotiation are completed.

[0568] Next, a camera built into the device (e.g., smart glasses or a head-mounted display) captures the faces of participants or customers in real time, and the captured image data is immediately sent to a cloud server.

[0569] Next, the cloud server receives the transmitted image data and analyzes it using an emotion analysis API (e.g., Google Cloud Vision API or Microsoft Azure Face API). At this time, the emotional state (e.g., joy, anxiety, anger, etc.) is determined and the results are stored in a database.

[0570] Based on the obtained emotional data, the server generates feedback in real time. For example, if the customer looks anxious, the server generates feedback such as "The customer is feeling anxious. He or she may need help." This feedback is immediately displayed on the device screen (e.g., smart glasses), allowing the user to take appropriate action.

[0571] Additionally, the server continuously evaluates sentiment data and generates alerts on the progress of sales negotiations and the probability of success of customer interactions, allowing additional measures to be taken at the appropriate time.

[0572] For example, if a customer looks anxious while browsing products in a virtual store, the smart glasses will display a notification saying, "The customer is anxious and may need help." This allows staff to respond quickly and provide efficient customer service.

[0573] The system also uses a natural language processing API to analyze participants' comments and constantly monitor for signs of harassment. If a comment is deemed inappropriate, it immediately displays a warning and sends an alert to an administrator.

[0574] An example of a prompt to input to a generative AI model is:

[0575] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[0576] In this way, the system integrates a variety of technologies to improve the quality of interactions and increase customer satisfaction in remote environments.

[0577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0578] Step 1: Start the meeting

[0579] A user launches an application and starts a meeting or business meeting. The server receives this information and creates a meeting or business meeting session. The input includes the user ID, meeting ID, and participant information, and the output is the start of a meeting session.

[0580] Step 2: Register participant information

[0581] The server registers information about each participant, including name, job title, user ID, etc. The input includes basic participant information, and the output generates participant information registered in the database.

[0582] Step 3: Capture image data

[0583] The device (such as smart glasses) captures the face of a participant or customer in real time using a built-in camera and generates image data. The input is a face image, and the output is the captured image data.

[0584] Step 4: Sending image data

[0585] The device sends the captured image data to the cloud server. The input is the captured image data, and the output is the image data sent to the server.

[0586] Step 5: Sentiment Analysis

[0587] The server sends the received image data to an emotion analysis API and obtains the analysis results. The input includes image data, and the emotional state is output as a result of facial recognition and emotion analysis. Specifically, Google Cloud Vision API and Microsoft Azure Face API are used.

[0588] Step 6: Storing Emotion Data

[0589] The server stores the analysis results in a database. The input is the emotion analysis results, and the output is the emotion data stored in the database.

[0590] Step 7: Generate feedback

[0591] The server generates real-time feedback based on the emotion data stored in the database. The input includes the analyzed emotion data, and the output is the generated feedback. For example, if the customer is feeling anxious, the generated feedback is "The customer is feeling anxious. He or she may need help."

[0592] Step 8: Notification of feedback

[0593] The terminal notifies the user of the generated feedback. The input is the generated feedback, and the output is a notification to the user. This notification is displayed, for example, as a pop-up on the display of the smart glasses.

[0594] Step 9: Analyzing Speech

[0595] The server analyzes participants' comments in real time to detect signs of harassment. The input includes audio data, and a natural language processing API is used to determine whether the comments are appropriate, and the analysis results are output.

[0596] Step 10: Harassment Warning

[0597] If the server detects signs of harassment, it immediately displays a warning and sends an alert to the administrator. The input is the analyzed speech data, and the output is a warning notification to the user and an alert to the administrator.

[0598] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[0599] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] The system of the present invention was developed to facilitate smooth communication in remote environments. This system can register participant information at the start of a meeting, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[0602] The main process of the system is as follows:

[0603] 1. Starting a meeting

[0604] (User) The user launches the conferencing application and starts the conference.

[0605] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0606] 2. Capture image data

[0607] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0608] (Terminal) Image and audio data captured in real time is sent to the server.

[0609] 3. Emotion analysis

[0610] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[0611] (Server) The emotion analysis API and emotion engine process the image and audio data, determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and return the results to the server. The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[0612] (Server) Save the analysis results in the database.

[0613] 4. Generate feedback

[0614] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0615] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0616] 5. Feedback Notification

[0617] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0618] (User) The user selects and executes the appropriate action from the presented measures.

[0619] 6. Managing the progress of business negotiations

[0620] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0621] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0622] 7. Harassment Detection

[0623] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0624] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0625] This system can facilitate smooth communication in a remote environment, improve the success rate of business negotiations, and prevent harassment from occurring. For example, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which allows the business negotiation to proceed smoothly. This makes it possible to achieve effective communication in a remote environment, just like face-to-face communication.

[0626] The processing flow will be explained below.

[0627] Step 1:

[0628] (User) The user launches the conferencing application and starts the conference.

[0629] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0630] Step 2:

[0631] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0632] (Terminal) Image and audio data captured in real time is sent to the server.

[0633] Step 3:

[0634] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[0635] (Server) The emotion analysis API and emotion engine process the image and audio data to determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.). The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[0636] (Server) Save the analysis results in the database.

[0637] Step 4:

[0638] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0639] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0640] Step 5:

[0641] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0642] (User) The user selects and executes the appropriate action from the presented measures.

[0643] Step 6:

[0644] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0645] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0646] Step 7:

[0647] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0648] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0649] Example 2

[0650] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0651] In remote meetings, there is a need to grasp participants' emotional states in real time, ensuring smooth business negotiations while preventing harassment. However, conventional systems lack sufficient emotional analysis and feedback generation, which can significantly impact the success of business negotiations. In particular, if appropriate measures are not provided when participants have doubts or concerns, communication efficiency decreases, affecting the success of business negotiations. Furthermore, if harassment is not detected and addressed early, the safety and security of participants in remote meetings may be threatened. To solve these issues, a system is needed that can accurately analyze participants' emotional states and quickly provide appropriate feedback.

[0652] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0653] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image and audio data of each participant, means for analyzing emotions from the captured image and audio data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for continuously evaluating the progress of the business negotiation and grasping the situation, means for analyzing the content of participants' comments and detecting whether the comments constitute harassment, and means for issuing warnings and alerts if harassment is detected. This allows for a real-time understanding of participants' emotional states in remote conferences and the provision of appropriate feedback and countermeasures, thereby facilitating smooth communication and improving the success of business negotiations. Furthermore, preventing harassment before it occurs ensures participants' peace of mind and safety.

[0654] "Participant information" refers to personal identification information such as user ID, name, and job title registered in the conference system.

[0655] "Image data" is digital data of still or moving images captured by a camera.

[0656] "Audio data" is digital data of audio captured by a microphone.

[0657] "Emotion analysis" is the process of determining the emotional state of participants based on image and audio data using specific algorithms or engines.

[0658] A "database" is a system for efficiently storing, retrieving, and managing large amounts of digital data.

[0659] "Feedback" refers to notifications or instructions to participants that are generated based on analyzed emotional data.

[0660] "Deal progress" refers to indicators or data for assessing whether a deal is progressing smoothly or heading towards success.

[0661] "Harassment detection" is the process of analyzing participants' comments and identifying whether they constitute inappropriate or harassing content.

[0662] A "warning" is a warning message that is displayed to participants when harassment is detected.

[0663] An "alert" is a warning that is sent to a system administrator when harassment is detected.

[0664] The system of the present invention was developed to facilitate smooth communication in remote environments. This system registers participant information at the start of a conference, captures image data of each participant, analyzes emotions from the captured image and audio data, and stores the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[0665] A specific embodiment for implementing the system is as follows.

[0666] Hardware and software used

[0667] 1. User device: A computer or smart device (e.g., PC, tablet, smartphone) with a camera and microphone.

[0668] 2. Server: An integrated management system equipped with a high-performance processor that performs emotion analysis and database operations.

[0669] 3. Conferencing application: Software that allows users to launch and manage meetings (e.g., Zoom, Microsoft Teams).

[0670] 4. Sentiment Analysis API: Cloud services for analyzing emotional states (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API).

[0671] Description of the main process

[0672] Starting a meeting

[0673] A user launches a conference application and starts a conference. The user clicks the "Start Conference" button and enters the conference details (title, date and time, participants, etc.).

[0674] The server receives the user's request, creates a conference session (e.g., a unique conference ID identified by a UUID), and saves the participant information from the user's input into a database.

[0675] Image data capture

[0676] The device activates the camera and captures the participant's face as a real-time video feed, capturing frames at regular intervals (e.g., 30 frames per second).

[0677] The device transmits video data captured in real time and audio data obtained from the microphone to the server in streaming format. Security is ensured by encrypting the transmitted data.

[0678] Emotion analysis

[0679] The server receives the video and audio data sent from the terminal and checks the consistency of the received data.

[0680] The server sends the received data to the sentiment analysis API and the sentiment engine, and performs preprocessing (e.g., face recognition and cropping) if necessary.

[0681] The emotion analysis API analyzes the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns the results to the server.

[0682] The server generates complex emotional data and stores the analysis results in a database.

[0683] Feedback generation and notification

[0684] The server queries the emotional data stored in the database and generates feedback if a specific emotional state is detected, e.g., "XXX is in doubt. You should take the following actions: 1. Answer the question 2. Provide further explanation 3. Accept the question."

[0685] The device receives the feedback and notifies the user, either via a pop-up on the screen or an audio alert.

[0686] The user selects and executes the appropriate action from the presented countermeasures.

[0687] Negotiation progress management

[0688] The server evaluates the emotional data in real time to understand the progress of the negotiation and calculates the balance between positive and negative emotions.

[0689] The server evaluates the chances of success of the deal and generates specific alerts depending on the specific situation, e.g. "Given your current emotional state, the chances of success of the deal are 70%."

[0690] Harassment Detection

[0691] The server analyzes participants' comments using a natural language processing API to identify whether they constitute inappropriate or harassing content.

[0692] If the server detects any inappropriate comments, it will immediately display a warning and send an alert to the administrator, for example, a warning message will be displayed on the device saying "That comment is inappropriate."

[0693] Examples of concrete examples and prompts

[0694] As an example, consider a situation where a user receives feedback during a business meeting saying, "YYY has a question." In this case, the user can take the measure of "accepting questions," allowing the business meeting to proceed smoothly. As a result, effective communication similar to face-to-face communication can be achieved even in a remote environment.

[0695] Prompt Sentence Examples

[0696] "Analyze participants' emotions during a meeting."

[0697] "Generate measures to take if person Y feels anxious."

[0698] "Calculate the probability of the meeting being successful."

[0699] In this way, this system facilitates communication in a remote environment, improves the chances of successful business negotiations, and prevents harassment from occurring.

[0700] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0701] Step 1:

[0702] Starting a meeting

[0703] The user starts the conference application and clicks the "Start Conference" button. Detailed information about the conference (title, date and time, participant information) is required as input.

[0704] The server receives the user's input information and creates a conference session. By saving the conference ID and participant information in a database, the output is the establishment of a new conference session and the registration of participant information.

[0705] Step 2:

[0706] Image data capture

[0707] The device activates the camera as soon as the meeting starts and captures participants' faces in real time, and requires a video feed from the camera as input.

[0708] The device continuously captures video frames in real time (e.g., 30 frames per second) and also receives audio data from the microphone. The output is the captured video and audio data.

[0709] Step 3:

[0710] Transmission of image data and audio data

[0711] The device encrypts the captured video and audio data and sends them to the server in streaming format. The data captured by the device is required as input.

[0712] The server receives these data and checks their integrity. As an output, the data whose integrity has been confirmed is stored on the server.

[0713] Step 4:

[0714] Emotion analysis

[0715] The server sends the received video and audio data to an emotion analysis API (e.g., Microsoft Azure Emotion API) and emotion engine. The received data is used as input.

[0716] The emotion analysis API analyzes the video and audio data, calculates the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns it to the server. The analysis results (emotional state and confidence level) are obtained as output.

[0717] The server generates composite emotion data and stores it in a database.

[0718] Step 5:

[0719] Generate feedback

[0720] The server queries the emotion data stored in the database and generates feedback if a specific emotional state (e.g., doubt, anxiety) is detected in real time. As input, it requires emotion data in the database.

[0721] The server generates feedback (e.g., "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept the question") as output.

[0722] Step 6:

[0723] Feedback Notification

[0724] The terminal notifies the user of the feedback received from the server. Feedback data is required as input.

[0725] The device will notify the user of the feedback via a pop-up on the screen or an audio alert, and the notified feedback will be available as output.

[0726] Step 7:

[0727] Negotiation progress management

[0728] The server continuously evaluates the emotion data and grasps the progress of the business negotiation in real time. Continuous emotion data is required as input.

[0729] The server evaluates the probability of success of the negotiation and generates a specific alert (e.g., "Based on your current emotional state, the probability of success of the negotiation is 70%) according to the progress. The progress management alert is obtained as an output.

[0730] Step 8:

[0731] Harassment Detection

[0732] The server analyzes participants' speech in real time using a natural language processing API. Voice data is required as input.

[0733] If the server detects harassing comments, it will immediately generate a warning and alert (e.g., "That comment is inappropriate"). The output will be a warning and an alert to the administrator.

[0734] These steps will ensure that remote meetings and business negotiations proceed smoothly, enable participants to understand their emotional state, and provide appropriate feedback. They will also help prevent harassment and create an environment where participants can feel safe and secure.

[0735] (Application example 2)

[0736] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] Many challenges remain in facilitating communication and improving the success rate of business negotiations in remote environments. In particular, in factory environments, it is difficult to grasp the emotional state of workers in real time, which impacts work efficiency and safety. Furthermore, early detection and countermeasures for harassment are insufficient, making it difficult to maintain a healthy work environment. A method is needed to overcome these challenges and improve worker safety and work efficiency.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0739] In this invention, the server includes: means for registering participant information at the start of a conference; means for capturing image data of each participant; means for analyzing emotions from the captured image data; means for storing the analyzed emotional data in a database; means for generating feedback in real time based on the emotional data; means for notifying participants of the generated feedback; means for analyzing the emotional state of factory workers in their work environment and providing real-time feedback; and means for detecting signs of harassment during work and sending an alert to a manager. This enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. Furthermore, it facilitates early detection of harassment and appropriate countermeasures.

[0740] A "meeting" is a place where multiple participants gather based on a specific purpose or agenda to share information, discuss, and make decisions.

[0741] "Participant information" is information that identifies and identifies people who participate in a meeting or work environment, such as name, job title, and user ID.

[0742] "Image data" refers to digital image files containing visual data captured by a camera or other device.

[0743] "Emotion analysis" is the process of analyzing the emotional state of participants or workers based on their facial expressions, voice, behavior, etc. from captured image data.

[0744] A "database" is a system for systematically storing and managing analyzed emotional data and other information.

[0745] "Feedback" refers to real-time information and advice provided to users based on emotional data.

[0746] A "factory worker" is a person who engages in production activities through mechanical or manual labor in a manufacturing environment.

[0747] The "work environment" refers to the place where workers perform their daily work, such as a factory or manufacturing site, and the surrounding environment.

[0748] "Real time" refers to a state in which processing and feedback are carried out simultaneously and immediately.

[0749] "Harassment" refers to inappropriate and offensive words, actions, or behavior in the work environment that have a negative impact on the health and work of workers.

[0750] A "warning" is a message that alerts users and administrators when inappropriate behavior such as harassment is detected.

[0751] A "system" is a comprehensive operational structure that integrates various means in a meeting or work environment.

[0752] This invention provides a system for analyzing the emotional states of workers in a factory environment in real time and providing efficient feedback. The system includes means for registering participant information at the start of a meeting, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for detecting signs of harassment during work and sending a warning to a manager.

[0753] The server registers participant information at the start of a meeting. This information includes user ID, name, and job title. This makes it possible to identify each participant. Each participant's device is equipped with a camera, which captures image data of the participant in real time. The captured image data is sent to the server, where emotion analysis is performed using EmotionAPI.

[0754] The results of emotion analysis are stored in a database. EmotionAPI works in conjunction with image processing libraries such as OpenCV to analyze facial expressions and vocal characteristics from captured image data. Feedback is generated in real time based on the results of this analysis. For example, if a worker feels doubtful or anxious, the server will generate feedback such as, "This worker has doubts. Please provide assistance."

[0755] The generated feedback is sent to the worker's device, and a similar notification is also sent to the manager. This allows the worker to receive appropriate support and improves work efficiency. The server also analyzes the content of comments made while working, and if inappropriate comments are detected, a warning is immediately displayed and an alert is sent to the manager. This enables early detection and countermeasures against harassment.

[0756] For example, when the system is started at the start of work, it automatically captures the worker's face and performs real-time emotional analysis using the Emotion API. If emotions such as "doubt" or "anxiety" are detected during work, the robot's display will display "This worker has doubts. Help is needed," and a notification will also be sent to the manager.

[0757] An example of a prompt statement can be written as follows:

[0758] Create a factory management system that combines image capture and emotion analysis. Include a function that analyzes the emotional state of workers, generates feedback when doubts or concerns are detected, and notifies workers and managers.

[0759] With this configuration, the present invention enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. It also facilitates early detection of harassment and the implementation of appropriate countermeasures.

[0760] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0761] Step 1:

[0762] The server registers participant information when a conference starts. Specifically, a user starts the conference application and inputs participant information such as user ID, name, and job title. This data is sent to the server and stored in a database. This makes it possible to identify participants. The input data is user ID, name, and job title, and the output data is the participant information stored in the database.

[0763] Step 2:

[0764] The device activates the camera and captures image data of each participant. The camera photographs the participants' faces in real time and sends the image data to the server. The input data is the photographed image data, and the output data is the image data sent to the server.

[0765] Step 3:

[0766] The server sends the received image data to the Emotion API for emotion analysis. The Emotion API uses OpenCV to analyze facial expressions and other features from the image data. The analysis results are output as emotional states such as joy, sadness, anger, surprise, fear, and calm. The input data is the image data, and the output data is the analyzed emotional state.

[0767] Step 4:

[0768] The server stores the analyzed emotional data in a database. The emotional states obtained as a result of the analysis (e.g., joy, sadness, anger, surprise, fear, calm, etc.) are stored in association with the corresponding participant information in the database. The input data is the result of the emotion analysis, and the output data is the emotional data stored in the database.

[0769] Step 5:

[0770] The server generates real-time feedback based on the emotion data stored in the database. For example, if the emotion data contains doubts or anxiety, the server generates feedback such as "This worker is feeling doubtful. Please provide assistance." The input data is the stored emotion data, and the output data is the generated feedback.

[0771] Step 6:

[0772] The generated feedback is notified to the device. The device notifies the user of the feedback content by displaying a pop-up or by audio alert. The user checks the notified feedback and takes appropriate action. The input data is the generated feedback, and the output data is the feedback notification.

[0773] Step 7:

[0774] The server analyzes what is said during work and monitors for signs of harassment. It uses a natural language processing API to analyze the content of speech, and if inappropriate behavior is detected, it generates warnings and alerts and notifies administrators. The input data is what the worker says, and the output data is the warnings and alerts that are the results of the analysis.

[0775] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0776] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0777] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0778] [Third embodiment]

[0779] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0780] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0781] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0782] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0783] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0784] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0785] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0786] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0787] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0788] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0789] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0790] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0791] The system of the present invention was developed to facilitate communication in remote environments. This system can register participant information at the start of a conference, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. Furthermore, it has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback.

[0792] The main process of the system is as follows:

[0793] 1. Starting a meeting

[0794] (User) The user launches the conferencing application and starts the conference.

[0795] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0796] 2. Capture image data

[0797] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0798] (Terminal) Sends the captured image data to the server in real time.

[0799] 3. Emotion analysis

[0800] (Server) The server sends the captured image data to an emotion analysis API (e.g., a general facial recognition API).

[0801] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[0802] (Server) Save the analysis results in the database.

[0803] 4. Generate feedback

[0804] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0805] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0806] 5. Feedback Notification

[0807] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0808] (User) The user selects and executes the appropriate action from the presented measures.

[0809] 6. Managing the progress of business negotiations

[0810] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0811] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0812] 7. Harassment Detection

[0813] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0814] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0815] Through these processes, the system of the present invention not only facilitates communication in remote environments and improves the success rate of business negotiations, but also prevents harassment from occurring. As a specific example, a situation can be imagined in which, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which will allow the business negotiation to proceed smoothly. This feedback function enables efficient communication in a remote environment, just like face-to-face communication.

[0816] The processing flow will be explained below.

[0817] Step 1:

[0818] (User) The user launches the conferencing application and starts the conference.

[0819] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0820] Step 2:

[0821] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0822] (Terminal) Sends the captured image data to the server in real time.

[0823] Step 3:

[0824] (Server) The server receives the captured image data and sends it to the emotion analysis API.

[0825] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[0826] (Server) Save the analysis results in the database.

[0827] Step 4:

[0828] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0829] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0830] Step 5:

[0831] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0832] (User) The user selects and executes the appropriate action from the presented measures.

[0833] Step 6:

[0834] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0835] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0836] Step 7:

[0837] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[0838] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[0839] Example 1

[0840] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0841] In today's remote meetings, physical distance often makes communication difficult, making it particularly difficult to recognize emotions and questions. It's also difficult to respond immediately when inappropriate remarks (harassment) are made, which reduces the productivity of meetings. To solve these issues, a system is needed that can analyze participants' emotions in real time and provide appropriate feedback.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0843] In this invention, the server includes means for registering participant information at the start of a conference, means for acquiring image data of each participant, means for analyzing emotions from the acquired image data, means for saving the analyzed emotional data, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for analyzing participants' speech content and detecting inappropriate remarks. This enables recognition of emotions and provision of feedback in real time in remote conferences, thereby enabling smoother communication and improved conference productivity.

[0844] "Participant information" refers to data used to identify and specify individuals, such as user ID, name, job title, and email address.

[0845] "Image data" refers to digital images of participants' faces and expressions.

[0846] "Analyzing emotions" refers to the process of determining an emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.) from the acquired image data.

[0847] A "database" is an information system for storing and managing analyzed emotion data.

[0848] "Feedback" refers to notifications (e.g., specific actions or countermeasures) that are generated for participants based on the analyzed emotional data.

[0849] "Notify" means communicating the generated feedback to participants in the form of an on-screen pop-up, chat message, audio alert, etc.

[0850] "Utterance content" is the set of words and sentences spoken by participants during a conference.

[0851] "Inappropriate comments" refers to harassment or comments that are ethically questionable.

[0852] "Warnings and Alerts" are warning messages or notifications that are displayed when inappropriate remarks are detected.

[0853] The system of this invention aims to facilitate communication in a remote environment. The system has the functions of registering participant information at the start of a conference, capturing image data of each participant, analyzing emotions from the captured image data, and saving the analysis results in a database. Furthermore, the system provides a function of generating feedback in real time based on the emotion data and notifying the participants of the generated feedback. It also has a function of analyzing the content of participants' speech and detecting inappropriate remarks.

[0854] The program for realizing this system uses the following hardware and software.

[0855] Hardware used

[0856] Server: Responsible for conference session generation, database management, sentiment analysis and feedback generation.

[0857] Device: A computer or smartphone used by a participant, equipped with a camera and microphone to capture image and audio data.

[0858] Database: Stores the analyzed emotion data and information about the meeting.

[0859] Software used

[0860] Sentiment Analysis API: An API for analyzing emotions from image data (e.g., Microsoft Azure Face API or Google Cloud Vision).

[0861] Natural language processing APIs: APIs for analyzing speech and detecting inappropriate language (e.g., Amazon Comprehend or IBM Watson Natural Language Understanding).

[0862] Processing flow

[0863] The main process of the system is as follows:

[0864] 1. Starting a meeting

[0865] A user launches a conferencing application and starts a conference.

[0866] The server generates a conference session, receives information about each participant, and registers it in a database.

[0867] 2. Capture and send image data

[0868] The device activates the camera and captures participants' faces in real time.

[0869] The image data captured by the terminal is periodically compressed and sent to the server.

[0870] 3. Sentiment analysis and data storage

[0871] The image data received by the server is sent to the emotion analysis API.

[0872] The emotion analysis API analyzes the image data, determines the emotional state, and returns the results to the server.

[0873] The server associates the analysis results with the corresponding user ID and stores them in a database.

[0874] 4. Feedback Generation and Notification

[0875] The server analyzes the emotion data stored in the database and generates feedback when a specific emotion is detected.

[0876] The device notifies the user of the generated feedback, such as via an on-screen pop-up or an audio alert.

[0877] 5. Analysis of speech content and warnings

[0878] The server sends the conversation content to a natural language processing API and analyzes it for inappropriate remarks.

[0879] If inappropriate remarks are detected, the server generates warnings and alerts and notifies the device.

[0880] Specific examples

[0881] For example, during a business meeting, if a user receives feedback such as "YYY has a question," the user can take action by "accepting questions," which will allow the business meeting to proceed smoothly. This allows for efficient communication in a remote environment, just like face-to-face communication.

[0882] Prompt Sentence Examples

[0883] Develop a system that analyzes participants' emotions during the negotiation and provides real-time feedback. For example, if a participant's image data indicates doubts or concerns, include a function that notifies the participant of this as feedback.

[0884] With the above specific examples and processing methods, this system can facilitate communication in remote environments, improve the success rate of business negotiations, and prevent harassment from occurring.

[0885] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0886] Step 1: Start the meeting

[0887] Enter: The user launches the conferencing application and enters their authentication information.

[0888] Action: The user clicks the "Start Meeting" button.

[0889] Server process: The server creates a conference session, receives information about each participant, and registers it in a database. The participant information includes the user ID, name, job title, and email address.

[0890] Output: The new conference session ID and participant information are saved in the database.

[0891] Step 2: Capture image data

[0892] Input: Conference session ID and each participant's device to activate the camera.

[0893] How it works: The device activates the camera and captures participants' faces in real time.

[0894] Terminal processing: The captured image data is compressed into JPEG format and sent to the server at regular intervals (e.g., every second).

[0895] Output: The compressed image data is sent to the server.

[0896] Step 3: Sentiment Analysis

[0897] Input: Image data sent from the device.

[0898] Server processing: The server sends the received image data to an emotion analysis API (e.g., Microsoft Azure Face API).

[0899] How it works: The emotion analysis API processes image data to determine the participant's emotional state (e.g., happy, sad, anger, surprise, fear, neutral, etc.).

[0900] Output: The sentiment analysis results are returned to the server.

[0901] Step 4: Storing emotion data

[0902] Input: Analysis results returned from the sentiment analysis API.

[0903] Server processing: The server associates the analysis results with the corresponding user ID and saves them in the database.

[0904] How it works: The server stores emotion data in a database.

[0905] Output: Emotion data is recorded in a database.

[0906] Step 5: Generate feedback

[0907] Input: Emotion data stored in a database.

[0908] Server processing: The server analyzes the emotion data and generates feedback if certain emotions (e.g., doubt, anxiety) are detected.

[0909] Action: The server generates feedback such as "XXX seems to be having difficulty understanding. Please consider the following actions: 1. Accept questions 2. Provide additional explanation 3. Answer questions."

[0910] Output: The generated feedback is sent to the device.

[0911] Step 6: Notification of feedback

[0912] Input: Feedback sent by the server.

[0913] Device action: The device receives the feedback and notifies the user with an on-screen popup, chat message, audio alert, etc.

[0914] Action: The device displays or sounds the feedback to the user.

[0915] Output: The user receives feedback.

[0916] Step 7: Analyzing the speech

[0917] Input: What participants say during the meeting.

[0918] Server processing: The server sends the utterance content to a natural language processing API (e.g., Amazon Comprehend).

[0919] How it works: A natural language processing API analyzes speech and detects inappropriate language.

[0920] Output: If inappropriate comments are detected, the analysis results are returned to the server.

[0921] Step 8: Notification of warnings and alerts

[0922] Input: Analysis results returned from the natural language processing API.

[0923] Server Processing: The server detects inappropriate comments and generates warnings and alerts.

[0924] What it does: The server generates a warning such as "That comment is inappropriate" and sends an alert to an administrator.

[0925] Output: Warnings and alerts are sent to the terminal and to the administrator.

[0926] These processing steps enable real-time emotion analysis and feedback during remote meetings, facilitating smoother communication and improving meeting productivity.

[0927] (Application example 1)

[0928] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0929] In remote communication, it is more difficult to grasp nonverbal emotions than in face-to-face conversations, which can hinder the progress of business negotiations, customer service, and other communication. It is also difficult to properly understand and quickly respond to feelings of anxiety and doubt during conversations with customers. Furthermore, harassment is on the rise in remote environments, making detection and response to it a critical issue.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0931] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for saving the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for capturing customer facial image data, means for analyzing customer emotions from the captured facial image data, means for saving the customer emotional data in a database, and means for notifying staff in real time of the customer's response based on the emotional data. This makes it possible to grasp the emotional state of customers even in a remote environment, and to provide feedback to increase the success rate of business negotiations and detect harassment.

[0932] "At the start of the meeting" refers to the moment when a remote meeting or business negotiation officially begins, and refers to the state in which the preparation stage is completed before that.

[0933] "Participant information" refers to details about each person attending a meeting or business meeting, including, for example, name, job title, user ID, etc.

[0934] "Image data" refers to still images and videos captured by an imaging device such as a camera.

[0935] "Emotion analysis" refers to the process of determining a person's emotional state from acquired image data using specific algorithms or APIs.

[0936] A "database" refers to a system or software that stores information in an organized manner and makes it easy to search and update.

[0937] "Feedback" refers to advice or information generated based on the analyzed emotional data and provided as a response to the user.

[0938] "Real-time" refers to a state in which processing and reaction occur almost instantaneously, meaning that there is very little time delay.

[0939] "Notifying participants" refers to the act of informing people participating in a meeting or business negotiation of information.

[0940] "Customer" means a person who visits the Virtual Store to purchase or use goods or services.

[0941] "Capturing" refers to obtaining image data using an imaging device such as a camera.

[0942] "Staff" refers to the person responsible for dealing with customers in the virtual store.

[0943] This invention is a system for facilitating communication primarily in remote environments, and provides the function of registering participant information at the start of a meeting or business negotiation, analyzing the emotions of each participant and customer in real time, and notifying them of the feedback. Specific programs and processing methods for realizing this invention are described below.

[0944] First, at the start of a meeting or business negotiation, the user launches an application, and the server generates a session for the meeting or business negotiation. Basic information about participants and customers is registered, and preparations for the meeting or business negotiation are completed.

[0945] Next, a camera built into the device (e.g., smart glasses or a head-mounted display) captures the faces of participants or customers in real time, and the captured image data is immediately sent to a cloud server.

[0946] Next, the cloud server receives the transmitted image data and analyzes it using an emotion analysis API (e.g., Google Cloud Vision API or Microsoft Azure Face API). At this time, the emotional state (e.g., joy, anxiety, anger, etc.) is determined and the results are stored in a database.

[0947] Based on the obtained emotional data, the server generates feedback in real time. For example, if the customer looks anxious, the server generates feedback such as "The customer is feeling anxious. He or she may need help." This feedback is immediately displayed on the device screen (e.g., smart glasses), allowing the user to take appropriate action.

[0948] Additionally, the server continuously evaluates sentiment data and generates alerts on the progress of sales negotiations and the probability of success of customer interactions, allowing additional measures to be taken at the appropriate time.

[0949] For example, if a customer looks anxious while browsing products in a virtual store, the smart glasses will display a notification saying, "The customer is anxious and may need help." This allows staff to respond quickly and provide efficient customer service.

[0950] The system also uses a natural language processing API to analyze participants' comments and constantly monitor for signs of harassment. If a comment is deemed inappropriate, it immediately displays a warning and sends an alert to an administrator.

[0951] An example of a prompt to input to a generative AI model is:

[0952] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[0953] In this way, the system integrates a variety of technologies to improve the quality of interactions and increase customer satisfaction in remote environments.

[0954] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0955] Step 1: Start the meeting

[0956] A user launches an application and starts a meeting or business meeting. The server receives this information and creates a meeting or business meeting session. The input includes the user ID, meeting ID, and participant information, and the output is the start of a meeting session.

[0957] Step 2: Register participant information

[0958] The server registers information about each participant, including name, job title, user ID, etc. The input includes basic participant information, and the output generates participant information registered in the database.

[0959] Step 3: Capture image data

[0960] The device (such as smart glasses) captures the face of a participant or customer in real time using a built-in camera and generates image data. The input is a face image, and the output is the captured image data.

[0961] Step 4: Sending image data

[0962] The device sends the captured image data to the cloud server. The input is the captured image data, and the output is the image data sent to the server.

[0963] Step 5: Sentiment Analysis

[0964] The server sends the received image data to an emotion analysis API and obtains the analysis results. The input includes image data, and the emotional state is output as a result of facial recognition and emotion analysis. Specifically, Google Cloud Vision API and Microsoft Azure Face API are used.

[0965] Step 6: Storing Emotion Data

[0966] The server stores the analysis results in a database. The input is the emotion analysis results, and the output is the emotion data stored in the database.

[0967] Step 7: Generate feedback

[0968] The server generates real-time feedback based on the emotion data stored in the database. The input includes the analyzed emotion data, and the output is the generated feedback. For example, if the customer is feeling anxious, the generated feedback is "The customer is feeling anxious. He or she may need help."

[0969] Step 8: Notification of feedback

[0970] The terminal notifies the user of the generated feedback. The input is the generated feedback, and the output is a notification to the user. This notification is displayed, for example, as a pop-up on the display of the smart glasses.

[0971] Step 9: Analyzing Speech

[0972] The server analyzes participants' comments in real time to detect signs of harassment. The input includes audio data, and a natural language processing API is used to determine whether the comments are appropriate, and the analysis results are output.

[0973] Step 10: Harassment Warning

[0974] If the server detects signs of harassment, it immediately displays a warning and sends an alert to the administrator. The input is the analyzed speech data, and the output is a warning notification to the user and an alert to the administrator.

[0975] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[0976] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[0977] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0978] The system of the present invention was developed to facilitate smooth communication in remote environments. This system can register participant information at the start of a meeting, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[0979] The main process of the system is as follows:

[0980] 1. Starting a meeting

[0981] (User) The user launches the conferencing application and starts the conference.

[0982] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[0983] 2. Capture image data

[0984] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[0985] (Terminal) Image and audio data captured in real time is sent to the server.

[0986] 3. Emotion analysis

[0987] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[0988] (Server) The emotion analysis API and emotion engine process the image and audio data, determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and return the results to the server. The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[0989] (Server) Save the analysis results in the database.

[0990] 4. Generate feedback

[0991] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[0992] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[0993] 5. Feedback Notification

[0994] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[0995] (User) The user selects and executes the appropriate action from the presented measures.

[0996] 6. Managing the progress of business negotiations

[0997] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[0998] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[0999] 7. Harassment Detection

[1000] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[1001] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[1002] This system can facilitate smooth communication in a remote environment, improve the success rate of business negotiations, and prevent harassment from occurring. For example, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which allows the business negotiation to proceed smoothly. This makes it possible to achieve effective communication in a remote environment, just like face-to-face communication.

[1003] The processing flow will be explained below.

[1004] Step 1:

[1005] (User) The user launches the conferencing application and starts the conference.

[1006] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[1007] Step 2:

[1008] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[1009] (Terminal) Image and audio data captured in real time is sent to the server.

[1010] Step 3:

[1011] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[1012] (Server) The emotion analysis API and emotion engine process the image and audio data to determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.). The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[1013] (Server) Save the analysis results in the database.

[1014] Step 4:

[1015] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[1016] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[1017] Step 5:

[1018] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[1019] (User) The user selects and executes the appropriate action from the presented measures.

[1020] Step 6:

[1021] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[1022] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[1023] Step 7:

[1024] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[1025] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[1026] Example 2

[1027] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1028] In remote meetings, there is a need to grasp participants' emotional states in real time, ensuring smooth business negotiations while preventing harassment. However, conventional systems lack sufficient emotional analysis and feedback generation, which can significantly impact the success of business negotiations. In particular, if appropriate measures are not provided when participants have doubts or concerns, communication efficiency decreases, affecting the success of business negotiations. Furthermore, if harassment is not detected and addressed early, the safety and security of participants in remote meetings may be threatened. To solve these issues, a system is needed that can accurately analyze participants' emotional states and quickly provide appropriate feedback.

[1029] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1030] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image and audio data of each participant, means for analyzing emotions from the captured image and audio data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for continuously evaluating the progress of the business negotiation and grasping the situation, means for analyzing the content of participants' comments and detecting whether the comments constitute harassment, and means for issuing warnings and alerts if harassment is detected. This allows for a real-time understanding of participants' emotional states in remote conferences and the provision of appropriate feedback and countermeasures, thereby facilitating smooth communication and improving the success of business negotiations. Furthermore, preventing harassment before it occurs ensures participants' peace of mind and safety.

[1031] "Participant information" refers to personal identification information such as user ID, name, and job title registered in the conference system.

[1032] "Image data" is digital data of still or moving images captured by a camera.

[1033] "Audio data" is digital data of audio captured by a microphone.

[1034] "Emotion analysis" is the process of determining the emotional state of participants based on image and audio data using specific algorithms or engines.

[1035] A "database" is a system for efficiently storing, retrieving, and managing large amounts of digital data.

[1036] "Feedback" refers to notifications or instructions to participants that are generated based on analyzed emotional data.

[1037] "Deal progress" refers to indicators or data for assessing whether a deal is progressing smoothly or heading towards success.

[1038] "Harassment detection" is the process of analyzing participants' comments and identifying whether they constitute inappropriate or harassing content.

[1039] A "warning" is a warning message that is displayed to participants when harassment is detected.

[1040] An "alert" is a warning that is sent to a system administrator when harassment is detected.

[1041] The system of the present invention was developed to facilitate smooth communication in remote environments. This system registers participant information at the start of a conference, captures image data of each participant, analyzes emotions from the captured image and audio data, and stores the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[1042] A specific embodiment for implementing the system is as follows.

[1043] Hardware and software used

[1044] 1. User device: A computer or smart device (e.g., PC, tablet, smartphone) with a camera and microphone.

[1045] 2. Server: An integrated management system equipped with a high-performance processor that performs emotion analysis and database operations.

[1046] 3. Conferencing application: Software that allows users to launch and manage meetings (e.g., Zoom, Microsoft Teams).

[1047] 4. Sentiment Analysis API: Cloud services for analyzing emotional states (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API).

[1048] Description of the main process

[1049] Starting a meeting

[1050] A user launches a conference application and starts a conference. The user clicks the "Start Conference" button and enters the conference details (title, date and time, participants, etc.).

[1051] The server receives the user's request, creates a conference session (e.g., a unique conference ID identified by a UUID), and saves the participant information from the user's input into a database.

[1052] Image data capture

[1053] The device activates the camera and captures the participant's face as a real-time video feed, capturing frames at regular intervals (e.g., 30 frames per second).

[1054] The device transmits video data captured in real time and audio data obtained from the microphone to the server in streaming format. Security is ensured by encrypting the transmitted data.

[1055] Emotion analysis

[1056] The server receives the video and audio data sent from the terminal and checks the consistency of the received data.

[1057] The server sends the received data to the sentiment analysis API and the sentiment engine, and performs preprocessing (e.g., face recognition and cropping) if necessary.

[1058] The emotion analysis API analyzes the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns the results to the server.

[1059] The server generates complex emotional data and stores the analysis results in a database.

[1060] Feedback generation and notification

[1061] The server queries the emotional data stored in the database and generates feedback if a specific emotional state is detected, e.g., "XXX is in doubt. You should take the following actions: 1. Answer the question 2. Provide further explanation 3. Accept the question."

[1062] The device receives the feedback and notifies the user, either via a pop-up on the screen or an audio alert.

[1063] The user selects and executes the appropriate action from the presented countermeasures.

[1064] Negotiation progress management

[1065] The server evaluates the emotional data in real time to understand the progress of the negotiation and calculates the balance between positive and negative emotions.

[1066] The server evaluates the chances of success of the deal and generates specific alerts depending on the specific situation, e.g. "Given your current emotional state, the chances of success of the deal are 70%."

[1067] Harassment Detection

[1068] The server analyzes participants' comments using a natural language processing API to identify whether they constitute inappropriate or harassing content.

[1069] If the server detects any inappropriate comments, it will immediately display a warning and send an alert to the administrator, for example, a warning message will be displayed on the device saying "That comment is inappropriate."

[1070] Examples of concrete examples and prompts

[1071] As an example, consider a situation where a user receives feedback during a business meeting saying, "YYY has a question." In this case, the user can take the measure of "accepting questions," allowing the business meeting to proceed smoothly. As a result, effective communication similar to face-to-face communication can be achieved even in a remote environment.

[1072] Prompt Sentence Examples

[1073] "Analyze participants' emotions during a meeting."

[1074] "Generate measures to take if person Y feels anxious."

[1075] "Calculate the probability of the meeting being successful."

[1076] In this way, this system facilitates communication in a remote environment, improves the chances of successful business negotiations, and prevents harassment from occurring.

[1077] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1078] Step 1:

[1079] Starting a meeting

[1080] The user starts the conference application and clicks the "Start Conference" button. Detailed information about the conference (title, date and time, participant information) is required as input.

[1081] The server receives the user's input information and creates a conference session. By saving the conference ID and participant information in a database, the output is the establishment of a new conference session and the registration of participant information.

[1082] Step 2:

[1083] Image data capture

[1084] The device activates the camera as soon as the meeting starts and captures participants' faces in real time, and requires a video feed from the camera as input.

[1085] The device continuously captures video frames in real time (e.g., 30 frames per second) and also receives audio data from the microphone. The output is the captured video and audio data.

[1086] Step 3:

[1087] Transmission of image data and audio data

[1088] The device encrypts the captured video and audio data and sends them to the server in streaming format. The data captured by the device is required as input.

[1089] The server receives these data and checks their integrity. As an output, the data whose integrity has been confirmed is stored on the server.

[1090] Step 4:

[1091] Emotion analysis

[1092] The server sends the received video and audio data to an emotion analysis API (e.g., Microsoft Azure Emotion API) and emotion engine. The received data is used as input.

[1093] The emotion analysis API analyzes the video and audio data, calculates the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns it to the server. The analysis results (emotional state and confidence level) are obtained as output.

[1094] The server generates composite emotion data and stores it in a database.

[1095] Step 5:

[1096] Generate feedback

[1097] The server queries the emotion data stored in the database and generates feedback if a specific emotional state (e.g., doubt, anxiety) is detected in real time. As input, it requires emotion data in the database.

[1098] The server generates feedback (e.g., "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept the question") as output.

[1099] Step 6:

[1100] Feedback Notification

[1101] The terminal notifies the user of the feedback received from the server. Feedback data is required as input.

[1102] The device will notify the user of the feedback via a pop-up on the screen or an audio alert, and the notified feedback will be available as output.

[1103] Step 7:

[1104] Negotiation progress management

[1105] The server continuously evaluates the emotion data and grasps the progress of the business negotiation in real time. Continuous emotion data is required as input.

[1106] The server evaluates the probability of success of the negotiation and generates a specific alert (e.g., "Based on your current emotional state, the probability of success of the negotiation is 70%) according to the progress. The progress management alert is obtained as an output.

[1107] Step 8:

[1108] Harassment Detection

[1109] The server analyzes participants' speech in real time using a natural language processing API. Voice data is required as input.

[1110] If the server detects harassing comments, it will immediately generate a warning and alert (e.g., "That comment is inappropriate"). The output will be a warning and an alert to the administrator.

[1111] These steps will ensure that remote meetings and business negotiations proceed smoothly, enable participants to understand their emotional state, and provide appropriate feedback. They will also help prevent harassment and create an environment where participants can feel safe and secure.

[1112] (Application example 2)

[1113] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] Many challenges remain in facilitating communication and improving the success rate of business negotiations in remote environments. In particular, in factory environments, it is difficult to grasp the emotional state of workers in real time, which impacts work efficiency and safety. Furthermore, early detection and countermeasures for harassment are insufficient, making it difficult to maintain a healthy work environment. A method is needed to overcome these challenges and improve worker safety and work efficiency.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1116] In this invention, the server includes: means for registering participant information at the start of a conference; means for capturing image data of each participant; means for analyzing emotions from the captured image data; means for storing the analyzed emotional data in a database; means for generating feedback in real time based on the emotional data; means for notifying participants of the generated feedback; means for analyzing the emotional state of factory workers in their work environment and providing real-time feedback; and means for detecting signs of harassment during work and sending an alert to a manager. This enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. Furthermore, it facilitates early detection of harassment and appropriate countermeasures.

[1117] A "meeting" is a place where multiple participants gather based on a specific purpose or agenda to share information, discuss, and make decisions.

[1118] "Participant information" is information that identifies and identifies people who participate in a meeting or work environment, such as name, job title, and user ID.

[1119] "Image data" refers to digital image files containing visual data captured by a camera or other device.

[1120] "Emotion analysis" is the process of analyzing the emotional state of participants or workers based on their facial expressions, voice, behavior, etc. from captured image data.

[1121] A "database" is a system for systematically storing and managing analyzed emotional data and other information.

[1122] "Feedback" refers to real-time information and advice provided to users based on emotional data.

[1123] A "factory worker" is a person who engages in production activities through mechanical or manual labor in a manufacturing environment.

[1124] The "work environment" refers to the place where workers perform their daily work, such as a factory or manufacturing site, and the surrounding environment.

[1125] "Real time" refers to a state in which processing and feedback are carried out simultaneously and immediately.

[1126] "Harassment" refers to inappropriate and offensive words, actions, or behavior in the work environment that have a negative impact on the health and work of workers.

[1127] A "warning" is a message that alerts users and administrators when inappropriate behavior such as harassment is detected.

[1128] A "system" is a comprehensive operational structure that integrates various means in a meeting or work environment.

[1129] This invention provides a system for analyzing the emotional states of workers in a factory environment in real time and providing efficient feedback. The system includes means for registering participant information at the start of a meeting, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for detecting signs of harassment during work and sending a warning to a manager.

[1130] The server registers participant information at the start of a meeting. This information includes user ID, name, and job title. This makes it possible to identify each participant. Each participant's device is equipped with a camera, which captures image data of the participant in real time. The captured image data is sent to the server, where emotion analysis is performed using EmotionAPI.

[1131] The results of emotion analysis are stored in a database. EmotionAPI works in conjunction with image processing libraries such as OpenCV to analyze facial expressions and vocal characteristics from captured image data. Feedback is generated in real time based on the results of this analysis. For example, if a worker feels doubtful or anxious, the server will generate feedback such as, "This worker has doubts. Please provide assistance."

[1132] The generated feedback is sent to the worker's device, and a similar notification is also sent to the manager. This allows the worker to receive appropriate support and improves work efficiency. The server also analyzes the content of comments made while working, and if inappropriate comments are detected, a warning is immediately displayed and an alert is sent to the manager. This enables early detection and countermeasures against harassment.

[1133] For example, when the system is started at the start of work, it automatically captures the worker's face and performs real-time emotional analysis using the Emotion API. If emotions such as "doubt" or "anxiety" are detected during work, the robot's display will display "This worker has doubts. Help is needed," and a notification will also be sent to the manager.

[1134] An example of a prompt statement can be written as follows:

[1135] Create a factory management system that combines image capture and emotion analysis. Include a function that analyzes the emotional state of workers, generates feedback when doubts or concerns are detected, and notifies workers and managers.

[1136] With this configuration, the present invention enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. It also facilitates early detection of harassment and the implementation of appropriate countermeasures.

[1137] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1138] Step 1:

[1139] The server registers participant information when a conference starts. Specifically, a user starts the conference application and inputs participant information such as user ID, name, and job title. This data is sent to the server and stored in a database. This makes it possible to identify participants. The input data is user ID, name, and job title, and the output data is the participant information stored in the database.

[1140] Step 2:

[1141] The device activates the camera and captures image data of each participant. The camera photographs the participants' faces in real time and sends the image data to the server. The input data is the photographed image data, and the output data is the image data sent to the server.

[1142] Step 3:

[1143] The server sends the received image data to the Emotion API for emotion analysis. The Emotion API uses OpenCV to analyze facial expressions and other features from the image data. The analysis results are output as emotional states such as joy, sadness, anger, surprise, fear, and calm. The input data is the image data, and the output data is the analyzed emotional state.

[1144] Step 4:

[1145] The server stores the analyzed emotional data in a database. The emotional states obtained as a result of the analysis (e.g., joy, sadness, anger, surprise, fear, calm, etc.) are stored in association with the corresponding participant information in the database. The input data is the result of the emotion analysis, and the output data is the emotional data stored in the database.

[1146] Step 5:

[1147] The server generates real-time feedback based on the emotion data stored in the database. For example, if the emotion data contains doubts or anxiety, the server generates feedback such as "This worker is feeling doubtful. Please provide assistance." The input data is the stored emotion data, and the output data is the generated feedback.

[1148] Step 6:

[1149] The generated feedback is notified to the device. The device notifies the user of the feedback content by displaying a pop-up or by audio alert. The user checks the notified feedback and takes appropriate action. The input data is the generated feedback, and the output data is the feedback notification.

[1150] Step 7:

[1151] The server analyzes what is said during work and monitors for signs of harassment. It uses a natural language processing API to analyze the content of speech, and if inappropriate behavior is detected, it generates warnings and alerts and notifies administrators. The input data is what the worker says, and the output data is the warnings and alerts that are the results of the analysis.

[1152] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1153] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1154] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1155] [Fourth embodiment]

[1156] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1157] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1158] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1159] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1160] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1161] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1162] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1163] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1164] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1165] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1166] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1167] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1168] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1169] The system of the present invention was developed to facilitate communication in remote environments. This system can register participant information at the start of a conference, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. Furthermore, it has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback.

[1170] The main process of the system is as follows:

[1171] 1. Starting a meeting

[1172] (User) The user launches the conferencing application and starts the conference.

[1173] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[1174] 2. Capture image data

[1175] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[1176] (Terminal) Sends the captured image data to the server in real time.

[1177] 3. Emotion analysis

[1178] (Server) The server sends the captured image data to an emotion analysis API (e.g., a general facial recognition API).

[1179] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[1180] (Server) Save the analysis results in the database.

[1181] 4. Generate feedback

[1182] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[1183] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[1184] 5. Feedback Notification

[1185] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[1186] (User) The user selects and executes the appropriate action from the presented measures.

[1187] 6. Managing the progress of business negotiations

[1188] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[1189] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[1190] 7. Harassment Detection

[1191] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[1192] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[1193] Through these processes, the system of the present invention not only facilitates communication in remote environments and improves the success rate of business negotiations, but also prevents harassment from occurring. As a specific example, a situation can be imagined in which, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which will allow the business negotiation to proceed smoothly. This feedback function enables efficient communication in a remote environment, just like face-to-face communication.

[1194] The processing flow will be explained below.

[1195] Step 1:

[1196] (User) The user launches the conferencing application and starts the conference.

[1197] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[1198] Step 2:

[1199] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[1200] (Terminal) Sends the captured image data to the server in real time.

[1201] Step 3:

[1202] (Server) The server receives the captured image data and sends it to the emotion analysis API.

[1203] (Server) The emotion analysis API processes the image data, determines the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and returns the results to the server.

[1204] (Server) Save the analysis results in the database.

[1205] Step 4:

[1206] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[1207] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[1208] Step 5:

[1209] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[1210] (User) The user selects and executes the appropriate action from the presented measures.

[1211] Step 6:

[1212] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[1213] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[1214] Step 7:

[1215] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[1216] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[1217] Example 1

[1218] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1219] In today's remote meetings, physical distance often makes communication difficult, making it particularly difficult to recognize emotions and questions. It's also difficult to respond immediately when inappropriate remarks (harassment) are made, which reduces the productivity of meetings. To solve these issues, a system is needed that can analyze participants' emotions in real time and provide appropriate feedback.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1221] In this invention, the server includes means for registering participant information at the start of a conference, means for acquiring image data of each participant, means for analyzing emotions from the acquired image data, means for saving the analyzed emotional data, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for analyzing participants' speech content and detecting inappropriate remarks. This enables recognition of emotions and provision of feedback in real time in remote conferences, thereby enabling smoother communication and improved conference productivity.

[1222] "Participant information" refers to data used to identify and specify individuals, such as user ID, name, job title, and email address.

[1223] "Image data" refers to digital images of participants' faces and expressions.

[1224] "Analyzing emotions" refers to the process of determining an emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.) from the acquired image data.

[1225] A "database" is an information system for storing and managing analyzed emotion data.

[1226] "Feedback" refers to notifications (e.g., specific actions or countermeasures) that are generated for participants based on the analyzed emotional data.

[1227] "Notify" means communicating the generated feedback to participants in the form of an on-screen pop-up, chat message, audio alert, etc.

[1228] "Utterance content" is the set of words and sentences spoken by participants during a conference.

[1229] "Inappropriate comments" refers to harassment or comments that are ethically questionable.

[1230] "Warnings and Alerts" are warning messages or notifications that are displayed when inappropriate remarks are detected.

[1231] The system of this invention aims to facilitate communication in a remote environment. The system has the functions of registering participant information at the start of a conference, capturing image data of each participant, analyzing emotions from the captured image data, and saving the analysis results in a database. Furthermore, the system provides a function of generating feedback in real time based on the emotion data and notifying the participants of the generated feedback. It also has a function of analyzing the content of participants' speech and detecting inappropriate remarks.

[1232] The program for realizing this system uses the following hardware and software.

[1233] Hardware used

[1234] Server: Responsible for conference session generation, database management, sentiment analysis and feedback generation.

[1235] Device: A computer or smartphone used by a participant, equipped with a camera and microphone to capture image and audio data.

[1236] Database: Stores the analyzed emotion data and information about the meeting.

[1237] Software used

[1238] Sentiment Analysis API: An API for analyzing emotions from image data (e.g., Microsoft Azure Face API or Google Cloud Vision).

[1239] Natural language processing APIs: APIs for analyzing speech and detecting inappropriate language (e.g., Amazon Comprehend or IBM Watson Natural Language Understanding).

[1240] Processing flow

[1241] The main process of the system is as follows:

[1242] 1. Starting a meeting

[1243] A user launches a conferencing application and starts a conference.

[1244] The server generates a conference session, receives information about each participant, and registers it in a database.

[1245] 2. Capture and send image data

[1246] The device activates the camera and captures participants' faces in real time.

[1247] The image data captured by the terminal is periodically compressed and sent to the server.

[1248] 3. Sentiment analysis and data storage

[1249] The image data received by the server is sent to the emotion analysis API.

[1250] The emotion analysis API analyzes the image data, determines the emotional state, and returns the results to the server.

[1251] The server associates the analysis results with the corresponding user ID and stores them in a database.

[1252] 4. Feedback Generation and Notification

[1253] The server analyzes the emotion data stored in the database and generates feedback when a specific emotion is detected.

[1254] The device notifies the user of the generated feedback, such as via an on-screen pop-up or an audio alert.

[1255] 5. Analysis of speech content and warnings

[1256] The server sends the conversation content to a natural language processing API and analyzes it for inappropriate remarks.

[1257] If inappropriate remarks are detected, the server generates warnings and alerts and notifies the device.

[1258] Specific examples

[1259] For example, during a business meeting, if a user receives feedback such as "YYY has a question," the user can take action by "accepting questions," which will allow the business meeting to proceed smoothly. This allows for efficient communication in a remote environment, just like face-to-face communication.

[1260] Prompt Sentence Examples

[1261] Develop a system that analyzes participants' emotions during the negotiation and provides real-time feedback. For example, if a participant's image data indicates doubts or concerns, include a function that notifies the participant of this as feedback.

[1262] With the above specific examples and processing methods, this system can facilitate communication in remote environments, improve the success rate of business negotiations, and prevent harassment from occurring.

[1263] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1264] Step 1: Start the meeting

[1265] Enter: The user launches the conferencing application and enters their authentication information.

[1266] Action: The user clicks the "Start Meeting" button.

[1267] Server process: The server creates a conference session, receives information about each participant, and registers it in a database. The participant information includes the user ID, name, job title, and email address.

[1268] Output: The new conference session ID and participant information are saved in the database.

[1269] Step 2: Capture image data

[1270] Input: Conference session ID and each participant's device to activate the camera.

[1271] How it works: The device activates the camera and captures participants' faces in real time.

[1272] Terminal processing: The captured image data is compressed into JPEG format and sent to the server at regular intervals (e.g., every second).

[1273] Output: The compressed image data is sent to the server.

[1274] Step 3: Sentiment Analysis

[1275] Input: Image data sent from the device.

[1276] Server processing: The server sends the received image data to an emotion analysis API (e.g., Microsoft Azure Face API).

[1277] How it works: The emotion analysis API processes image data to determine the participant's emotional state (e.g., happy, sad, anger, surprise, fear, neutral, etc.).

[1278] Output: The sentiment analysis results are returned to the server.

[1279] Step 4: Storing emotion data

[1280] Input: Analysis results returned from the sentiment analysis API.

[1281] Server processing: The server associates the analysis results with the corresponding user ID and saves them in the database.

[1282] How it works: The server stores emotion data in a database.

[1283] Output: Emotion data is recorded in a database.

[1284] Step 5: Generate feedback

[1285] Input: Emotion data stored in a database.

[1286] Server processing: The server analyzes the emotion data and generates feedback if certain emotions (e.g., doubt, anxiety) are detected.

[1287] Action: The server generates feedback such as "XXX seems to be having difficulty understanding. Please consider the following actions: 1. Accept questions 2. Provide additional explanation 3. Answer questions."

[1288] Output: The generated feedback is sent to the device.

[1289] Step 6: Notification of feedback

[1290] Input: Feedback sent by the server.

[1291] Device action: The device receives the feedback and notifies the user with an on-screen popup, chat message, audio alert, etc.

[1292] Action: The device displays or sounds the feedback to the user.

[1293] Output: The user receives feedback.

[1294] Step 7: Analyzing the speech

[1295] Input: What participants say during the meeting.

[1296] Server processing: The server sends the utterance content to a natural language processing API (e.g., Amazon Comprehend).

[1297] How it works: A natural language processing API analyzes speech and detects inappropriate language.

[1298] Output: If inappropriate comments are detected, the analysis results are returned to the server.

[1299] Step 8: Notification of warnings and alerts

[1300] Input: Analysis results returned from the natural language processing API.

[1301] Server Processing: The server detects inappropriate comments and generates warnings and alerts.

[1302] What it does: The server generates a warning such as "That comment is inappropriate" and sends an alert to an administrator.

[1303] Output: Warnings and alerts are sent to the terminal and to the administrator.

[1304] These processing steps enable real-time emotion analysis and feedback during remote meetings, facilitating smoother communication and improving meeting productivity.

[1305] (Application example 1)

[1306] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1307] In remote communication, it is more difficult to grasp nonverbal emotions than in face-to-face conversations, which can hinder the progress of business negotiations, customer service, and other communication. It is also difficult to properly understand and quickly respond to feelings of anxiety and doubt during conversations with customers. Furthermore, harassment is on the rise in remote environments, making detection and response to it a critical issue.

[1308] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1309] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for saving the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for capturing customer facial image data, means for analyzing customer emotions from the captured facial image data, means for saving the customer emotional data in a database, and means for notifying staff in real time of the customer's response based on the emotional data. This makes it possible to grasp the emotional state of customers even in a remote environment, and to provide feedback to increase the success rate of business negotiations and detect harassment.

[1310] "At the start of the meeting" refers to the moment when a remote meeting or business negotiation officially begins, and refers to the state in which the preparation stage is completed before that.

[1311] "Participant information" refers to details about each person attending a meeting or business meeting, including, for example, name, job title, user ID, etc.

[1312] "Image data" refers to still images and videos captured by an imaging device such as a camera.

[1313] "Emotion analysis" refers to the process of determining a person's emotional state from acquired image data using specific algorithms or APIs.

[1314] A "database" refers to a system or software that stores information in an organized manner and makes it easy to search and update.

[1315] "Feedback" refers to advice or information generated based on the analyzed emotional data and provided as a response to the user.

[1316] "Real-time" refers to a state in which processing and reaction occur almost instantaneously, meaning that there is very little time delay.

[1317] "Notifying participants" refers to the act of informing people participating in a meeting or business negotiation of information.

[1318] "Customer" means a person who visits the Virtual Store to purchase or use goods or services.

[1319] "Capturing" refers to obtaining image data using an imaging device such as a camera.

[1320] "Staff" refers to the person responsible for dealing with customers in the virtual store.

[1321] This invention is a system for facilitating communication primarily in remote environments, and provides the function of registering participant information at the start of a meeting or business negotiation, analyzing the emotions of each participant and customer in real time, and notifying them of the feedback. Specific programs and processing methods for realizing this invention are described below.

[1322] First, at the start of a meeting or business negotiation, the user launches an application, and the server generates a session for the meeting or business negotiation. Basic information about participants and customers is registered, and preparations for the meeting or business negotiation are completed.

[1323] Next, a camera built into the device (e.g., smart glasses or a head-mounted display) captures the faces of participants or customers in real time, and the captured image data is immediately sent to a cloud server.

[1324] Next, the cloud server receives the transmitted image data and analyzes it using an emotion analysis API (e.g., Google Cloud Vision API or Microsoft Azure Face API). At this time, the emotional state (e.g., joy, anxiety, anger, etc.) is determined and the results are stored in a database.

[1325] Based on the obtained emotional data, the server generates feedback in real time. For example, if the customer looks anxious, the server generates feedback such as "The customer is feeling anxious. He or she may need help." This feedback is immediately displayed on the device screen (e.g., smart glasses), allowing the user to take appropriate action.

[1326] Additionally, the server continuously evaluates sentiment data and generates alerts on the progress of sales negotiations and the probability of success of customer interactions, allowing additional measures to be taken at the appropriate time.

[1327] For example, if a customer looks anxious while browsing products in a virtual store, the smart glasses will display a notification saying, "The customer is anxious and may need help." This allows staff to respond quickly and provide efficient customer service.

[1328] The system also uses a natural language processing API to analyze participants' comments and constantly monitor for signs of harassment. If a comment is deemed inappropriate, it immediately displays a warning and sends an alert to an administrator.

[1329] An example of a prompt to input to a generative AI model is:

[1330] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[1331] In this way, the system integrates a variety of technologies to improve the quality of interactions and increase customer satisfaction in remote environments.

[1332] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1333] Step 1: Start the meeting

[1334] A user launches an application and starts a meeting or business meeting. The server receives this information and creates a meeting or business meeting session. The input includes the user ID, meeting ID, and participant information, and the output is the start of a meeting session.

[1335] Step 2: Register participant information

[1336] The server registers information about each participant, including name, job title, user ID, etc. The input includes basic participant information, and the output generates participant information registered in the database.

[1337] Step 3: Capture image data

[1338] The device (such as smart glasses) captures the face of a participant or customer in real time using a built-in camera and generates image data. The input is a face image, and the output is the captured image data.

[1339] Step 4: Sending image data

[1340] The device sends the captured image data to the cloud server. The input is the captured image data, and the output is the image data sent to the server.

[1341] Step 5: Sentiment Analysis

[1342] The server sends the received image data to an emotion analysis API and obtains the analysis results. The input includes image data, and the emotional state is output as a result of facial recognition and emotion analysis. Specifically, Google Cloud Vision API and Microsoft Azure Face API are used.

[1343] Step 6: Storing Emotion Data

[1344] The server stores the analysis results in a database. The input is the emotion analysis results, and the output is the emotion data stored in the database.

[1345] Step 7: Generate feedback

[1346] The server generates real-time feedback based on the emotion data stored in the database. The input includes the analyzed emotion data, and the output is the generated feedback. For example, if the customer is feeling anxious, the generated feedback is "The customer is feeling anxious. He or she may need help."

[1347] Step 8: Notification of feedback

[1348] The terminal notifies the user of the generated feedback. The input is the generated feedback, and the output is a notification to the user. This notification is displayed, for example, as a pop-up on the display of the smart glasses.

[1349] Step 9: Analyzing Speech

[1350] The server analyzes participants' comments in real time to detect signs of harassment. The input includes audio data, and a natural language processing API is used to determine whether the comments are appropriate, and the analysis results are output.

[1351] Step 10: Harassment Warning

[1352] If the server detects signs of harassment, it immediately displays a warning and sends an alert to the administrator. The input is the analyzed speech data, and the output is a warning notification to the user and an alert to the administrator.

[1353] As a concrete example, the prompt sentence to be input to the generative AI model is as follows:

[1354] "Customers are browsing products in a virtual store. Analyze their emotions based on facial image data and provide real-time feedback."

[1355] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1356] The system of the present invention was developed to facilitate smooth communication in remote environments. This system can register participant information at the start of a meeting, capture image data of each participant, analyze emotions from the captured image data, and store the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[1357] The main process of the system is as follows:

[1358] 1. Starting a meeting

[1359] (User) The user launches the conferencing application and starts the conference.

[1360] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[1361] 2. Capture image data

[1362] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[1363] (Terminal) Image and audio data captured in real time is sent to the server.

[1364] 3. Emotion analysis

[1365] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[1366] (Server) The emotion analysis API and emotion engine process the image and audio data, determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.), and return the results to the server. The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[1367] (Server) Save the analysis results in the database.

[1368] 4. Generate feedback

[1369] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[1370] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[1371] 5. Feedback Notification

[1372] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[1373] (User) The user selects and executes the appropriate action from the presented measures.

[1374] 6. Managing the progress of business negotiations

[1375] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[1376] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[1377] 7. Harassment Detection

[1378] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[1379] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[1380] This system can facilitate smooth communication in a remote environment, improve the success rate of business negotiations, and prevent harassment from occurring. For example, during a business negotiation, feedback such as "YYY has a question" is notified to the user, and the user can take the measure of "accepting questions," which allows the business negotiation to proceed smoothly. This makes it possible to achieve effective communication in a remote environment, just like face-to-face communication.

[1381] The processing flow will be explained below.

[1382] Step 1:

[1383] (User) The user launches the conferencing application and starts the conference.

[1384] (Server) The server creates a conference session and registers information about each participant, including their user ID, name, and job title.

[1385] Step 2:

[1386] (Device) Each participant's device activates a camera to capture the participant's face in real time.

[1387] (Terminal) Image and audio data captured in real time is sent to the server.

[1388] Step 3:

[1389] (Server) The server receives the captured image and audio data and sends it to the emotion analysis API and emotion engine.

[1390] (Server) The emotion analysis API and emotion engine process the image and audio data to determine the participant's emotional state (e.g., joy, sadness, anger, surprise, fear, calm, etc.). The emotion engine recognizes multiple emotional states simultaneously and generates composite emotion data.

[1391] (Server) Save the analysis results in the database.

[1392] Step 4:

[1393] (Server) Analyzes the emotion data stored in the database and generates feedback when a specific emotion (e.g., doubt or anxiety) is detected.

[1394] (Server) The generated feedback includes specific measures, such as "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept questions."

[1395] Step 5:

[1396] (Device) When feedback is generated, the device notifies the user, for example, via a pop-up on the screen or an audio alert.

[1397] (User) The user selects and executes the appropriate action from the presented measures.

[1398] Step 6:

[1399] (Server) Emotion data is continuously evaluated to understand the progress of the negotiation. If there is an imbalance between positive and negative emotions or if the success of the negotiation is in jeopardy, this is also notified to the user as feedback.

[1400] (Server) For example, generate a specific alert such as, "Based on your current emotional state, the chances of the deal succeeding are 70%."

[1401] Step 7:

[1402] (Server) Analyze participants' comments in real time using a natural language processing API and monitor for signs of harassment.

[1403] (Server) If an inappropriate comment is detected, a warning will be displayed immediately and an alert will be sent to the administrator. For example, a warning such as "That comment is inappropriate" will be displayed on the terminal.

[1404] Example 2

[1405] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1406] In remote meetings, there is a need to grasp participants' emotional states in real time, ensuring smooth business negotiations while preventing harassment. However, conventional systems lack sufficient emotional analysis and feedback generation, which can significantly impact the success of business negotiations. In particular, if appropriate measures are not provided when participants have doubts or concerns, communication efficiency decreases, affecting the success of business negotiations. Furthermore, if harassment is not detected and addressed early, the safety and security of participants in remote meetings may be threatened. To solve these issues, a system is needed that can accurately analyze participants' emotional states and quickly provide appropriate feedback.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1408] In this invention, the server includes means for registering participant information at the start of a conference, means for capturing image and audio data of each participant, means for analyzing emotions from the captured image and audio data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, means for continuously evaluating the progress of the business negotiation and grasping the situation, means for analyzing the content of participants' comments and detecting whether the comments constitute harassment, and means for issuing warnings and alerts if harassment is detected. This allows for a real-time understanding of participants' emotional states in remote conferences and the provision of appropriate feedback and countermeasures, thereby facilitating smooth communication and improving the success of business negotiations. Furthermore, preventing harassment before it occurs ensures participants' peace of mind and safety.

[1409] "Participant information" refers to personal identification information such as user ID, name, and job title registered in the conference system.

[1410] "Image data" is digital data of still or moving images captured by a camera.

[1411] "Audio data" is digital data of audio captured by a microphone.

[1412] "Emotion analysis" is the process of determining the emotional state of participants based on image and audio data using specific algorithms or engines.

[1413] A "database" is a system for efficiently storing, retrieving, and managing large amounts of digital data.

[1414] "Feedback" refers to notifications or instructions to participants that are generated based on analyzed emotional data.

[1415] "Deal progress" refers to indicators or data for assessing whether a deal is progressing smoothly or heading towards success.

[1416] "Harassment detection" is the process of analyzing participants' comments and identifying whether they constitute inappropriate or harassing content.

[1417] A "warning" is a warning message that is displayed to participants when harassment is detected.

[1418] An "alert" is a warning that is sent to a system administrator when harassment is detected.

[1419] The system of the present invention was developed to facilitate smooth communication in remote environments. This system registers participant information at the start of a conference, captures image data of each participant, analyzes emotions from the captured image and audio data, and stores the analyzed emotional data in a database. It also has the function of generating feedback in real time based on the emotional data and notifying the participants of the generated feedback. Furthermore, by combining it with an emotion engine for recognizing user emotions, it is possible to generate complex emotional data and distinguish them with high accuracy.

[1420] A specific embodiment for implementing the system is as follows.

[1421] Hardware and software used

[1422] 1. User device: A computer or smart device (e.g., PC, tablet, smartphone) with a camera and microphone.

[1423] 2. Server: An integrated management system equipped with a high-performance processor that performs emotion analysis and database operations.

[1424] 3. Conferencing application: Software that allows users to launch and manage meetings (e.g., Zoom, Microsoft Teams).

[1425] 4. Sentiment Analysis API: Cloud services for analyzing emotional states (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API).

[1426] Description of the main process

[1427] Starting a meeting

[1428] A user launches a conference application and starts a conference. The user clicks the "Start Conference" button and enters the conference details (title, date and time, participants, etc.).

[1429] The server receives the user's request, creates a conference session (e.g., a unique conference ID identified by a UUID), and saves the participant information from the user's input into a database.

[1430] Image data capture

[1431] The device activates the camera and captures the participant's face as a real-time video feed, capturing frames at regular intervals (e.g., 30 frames per second).

[1432] The device transmits video data captured in real time and audio data obtained from the microphone to the server in streaming format. Security is ensured by encrypting the transmitted data.

[1433] Emotion analysis

[1434] The server receives the video and audio data sent from the terminal and checks the consistency of the received data.

[1435] The server sends the received data to the sentiment analysis API and the sentiment engine, and performs preprocessing (e.g., face recognition and cropping) if necessary.

[1436] The emotion analysis API analyzes the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns the results to the server.

[1437] The server generates complex emotional data and stores the analysis results in a database.

[1438] Feedback generation and notification

[1439] The server queries the emotional data stored in the database and generates feedback if a specific emotional state is detected, e.g., "XXX is in doubt. You should take the following actions: 1. Answer the question 2. Provide further explanation 3. Accept the question."

[1440] The device receives the feedback and notifies the user, either via a pop-up on the screen or an audio alert.

[1441] The user selects and executes the appropriate action from the presented countermeasures.

[1442] Negotiation progress management

[1443] The server evaluates the emotional data in real time to understand the progress of the negotiation and calculates the balance between positive and negative emotions.

[1444] The server evaluates the chances of success of the deal and generates specific alerts depending on the specific situation, e.g. "Given your current emotional state, the chances of success of the deal are 70%."

[1445] Harassment Detection

[1446] The server analyzes participants' comments using a natural language processing API to identify whether they constitute inappropriate or harassing content.

[1447] If the server detects any inappropriate comments, it will immediately display a warning and send an alert to the administrator, for example, a warning message will be displayed on the device saying "That comment is inappropriate."

[1448] Examples of concrete examples and prompts

[1449] As an example, consider a situation where a user receives feedback during a business meeting saying, "YYY has a question." In this case, the user can take the measure of "accepting questions," allowing the business meeting to proceed smoothly. As a result, effective communication similar to face-to-face communication can be achieved even in a remote environment.

[1450] Prompt Sentence Examples

[1451] "Analyze participants' emotions during a meeting."

[1452] "Generate measures to take if person Y feels anxious."

[1453] "Calculate the probability of the meeting being successful."

[1454] In this way, this system facilitates communication in a remote environment, improves the chances of successful business negotiations, and prevents harassment from occurring.

[1455] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1456] Step 1:

[1457] Starting a meeting

[1458] The user starts the conference application and clicks the "Start Conference" button. Detailed information about the conference (title, date and time, participant information) is required as input.

[1459] The server receives the user's input information and creates a conference session. By saving the conference ID and participant information in a database, the output is the establishment of a new conference session and the registration of participant information.

[1460] Step 2:

[1461] Image data capture

[1462] The device activates the camera as soon as the meeting starts and captures participants' faces in real time, and requires a video feed from the camera as input.

[1463] The device continuously captures video frames in real time (e.g., 30 frames per second) and also receives audio data from the microphone. The output is the captured video and audio data.

[1464] Step 3:

[1465] Transmission of image data and audio data

[1466] The device encrypts the captured video and audio data and sends them to the server in streaming format. The data captured by the device is required as input.

[1467] The server receives these data and checks their integrity. As an output, the data whose integrity has been confirmed is stored on the server.

[1468] Step 4:

[1469] Emotion analysis

[1470] The server sends the received video and audio data to an emotion analysis API (e.g., Microsoft Azure Emotion API) and emotion engine. The received data is used as input.

[1471] The emotion analysis API analyzes the video and audio data, calculates the emotional state (e.g., joy, sadness, anger, etc.) and its confidence level, and returns it to the server. The analysis results (emotional state and confidence level) are obtained as output.

[1472] The server generates composite emotion data and stores it in a database.

[1473] Step 5:

[1474] Generate feedback

[1475] The server queries the emotion data stored in the database and generates feedback if a specific emotional state (e.g., doubt, anxiety) is detected in real time. As input, it requires emotion data in the database.

[1476] The server generates feedback (e.g., "XXX has a question. You should take the following actions: 1. Answer the question. 2. Provide additional explanation. 3. Accept the question") as output.

[1477] Step 6:

[1478] Feedback Notification

[1479] The terminal notifies the user of the feedback received from the server. Feedback data is required as input.

[1480] The device will notify the user of the feedback via a pop-up on the screen or an audio alert, and the notified feedback will be available as output.

[1481] Step 7:

[1482] Negotiation progress management

[1483] The server continuously evaluates the emotion data and grasps the progress of the business negotiation in real time. Continuous emotion data is required as input.

[1484] The server evaluates the probability of success of the negotiation and generates a specific alert (e.g., "Based on your current emotional state, the probability of success of the negotiation is 70%) according to the progress. The progress management alert is obtained as an output.

[1485] Step 8:

[1486] Harassment Detection

[1487] The server analyzes participants' speech in real time using a natural language processing API. Voice data is required as input.

[1488] If the server detects harassing comments, it will immediately generate a warning and alert (e.g., "That comment is inappropriate"). The output will be a warning and an alert to the administrator.

[1489] These steps will ensure that remote meetings and business negotiations proceed smoothly, enable participants to understand their emotional state, and provide appropriate feedback. They will also help prevent harassment and create an environment where participants can feel safe and secure.

[1490] (Application example 2)

[1491] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1492] Many challenges remain in facilitating communication and improving the success rate of business negotiations in remote environments. In particular, in factory environments, it is difficult to grasp the emotional state of workers in real time, which impacts work efficiency and safety. Furthermore, early detection and countermeasures for harassment are insufficient, making it difficult to maintain a healthy work environment. A method is needed to overcome these challenges and improve worker safety and work efficiency.

[1493] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1494] In this invention, the server includes: means for registering participant information at the start of a conference; means for capturing image data of each participant; means for analyzing emotions from the captured image data; means for storing the analyzed emotional data in a database; means for generating feedback in real time based on the emotional data; means for notifying participants of the generated feedback; means for analyzing the emotional state of factory workers in their work environment and providing real-time feedback; and means for detecting signs of harassment during work and sending an alert to a manager. This enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. Furthermore, it facilitates early detection of harassment and appropriate countermeasures.

[1495] A "meeting" is a place where multiple participants gather based on a specific purpose or agenda to share information, discuss, and make decisions.

[1496] "Participant information" is information that identifies and identifies people who participate in a meeting or work environment, such as name, job title, and user ID.

[1497] "Image data" refers to digital image files containing visual data captured by a camera or other device.

[1498] "Emotion analysis" is the process of analyzing the emotional state of participants or workers based on their facial expressions, voice, behavior, etc. from captured image data.

[1499] A "database" is a system for systematically storing and managing analyzed emotional data and other information.

[1500] "Feedback" refers to real-time information and advice provided to users based on emotional data.

[1501] A "factory worker" is a person who engages in production activities through mechanical or manual labor in a manufacturing environment.

[1502] The "work environment" refers to the place where workers perform their daily work, such as a factory or manufacturing site, and the surrounding environment.

[1503] "Real time" refers to a state in which processing and feedback are carried out simultaneously and immediately.

[1504] "Harassment" refers to inappropriate and offensive words, actions, or behavior in the work environment that have a negative impact on the health and work of workers.

[1505] A "warning" is a message that alerts users and administrators when inappropriate behavior such as harassment is detected.

[1506] A "system" is a comprehensive operational structure that integrates various means in a meeting or work environment.

[1507] This invention provides a system for analyzing the emotional states of workers in a factory environment in real time and providing efficient feedback. The system includes means for registering participant information at the start of a meeting, means for capturing image data of each participant, means for analyzing emotions from the captured image data, means for storing the analyzed emotional data in a database, means for generating feedback in real time based on the emotional data, means for notifying participants of the generated feedback, and means for detecting signs of harassment during work and sending a warning to a manager.

[1508] The server registers participant information at the start of a meeting. This information includes user ID, name, and job title. This makes it possible to identify each participant. Each participant's device is equipped with a camera, which captures image data of the participant in real time. The captured image data is sent to the server, where emotion analysis is performed using EmotionAPI.

[1509] The results of emotion analysis are stored in a database. EmotionAPI works in conjunction with image processing libraries such as OpenCV to analyze facial expressions and vocal characteristics from captured image data. Feedback is generated in real time based on the results of this analysis. For example, if a worker feels doubtful or anxious, the server will generate feedback such as, "This worker has doubts. Please provide assistance."

[1510] The generated feedback is sent to the worker's device, and a similar notification is also sent to the manager. This allows the worker to receive appropriate support and improves work efficiency. The server also analyzes the content of comments made while working, and if inappropriate comments are detected, a warning is immediately displayed and an alert is sent to the manager. This enables early detection and countermeasures against harassment.

[1511] For example, when the system is started at the start of work, it automatically captures the worker's face and performs real-time emotional analysis using the Emotion API. If emotions such as "doubt" or "anxiety" are detected during work, the robot's display will display "This worker has doubts. Help is needed," and a notification will also be sent to the manager.

[1512] An example of a prompt statement can be written as follows:

[1513] Create a factory management system that combines image capture and emotion analysis. Include a function that analyzes the emotional state of workers, generates feedback when doubts or concerns are detected, and notifies workers and managers.

[1514] With this configuration, the present invention enables real-time understanding of the emotional state of workers in a factory environment, improving work efficiency and safety. It also facilitates early detection of harassment and the implementation of appropriate countermeasures.

[1515] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1516] Step 1:

[1517] The server registers participant information when a conference starts. Specifically, a user starts the conference application and inputs participant information such as user ID, name, and job title. This data is sent to the server and stored in a database. This makes it possible to identify participants. The input data is user ID, name, and job title, and the output data is the participant information stored in the database.

[1518] Step 2:

[1519] The device activates the camera and captures image data of each participant. The camera photographs the participants' faces in real time and sends the image data to the server. The input data is the photographed image data, and the output data is the image data sent to the server.

[1520] Step 3:

[1521] The server sends the received image data to the Emotion API for emotion analysis. The Emotion API uses OpenCV to analyze facial expressions and other features from the image data. The analysis results are output as emotional states such as joy, sadness, anger, surprise, fear, and calm. The input data is the image data, and the output data is the analyzed emotional state.

[1522] Step 4:

[1523] The server stores the analyzed emotional data in a database. The emotional states obtained as a result of the analysis (e.g., joy, sadness, anger, surprise, fear, calm, etc.) are stored in association with the corresponding participant information in the database. The input data is the result of the emotion analysis, and the output data is the emotional data stored in the database.

[1524] Step 5:

[1525] The server generates real-time feedback based on the emotion data stored in the database. For example, if the emotion data contains doubts or anxiety, the server generates feedback such as "This worker is feeling doubtful. Please provide assistance." The input data is the stored emotion data, and the output data is the generated feedback.

[1526] Step 6:

[1527] The generated feedback is notified to the device. The device notifies the user of the feedback content by displaying a pop-up or by audio alert. The user checks the notified feedback and takes appropriate action. The input data is the generated feedback, and the output data is the feedback notification.

[1528] Step 7:

[1529] The server analyzes what is said during work and monitors for signs of harassment. It uses a natural language processing API to analyze the content of speech, and if inappropriate behavior is detected, it generates warnings and alerts and notifies administrators. The input data is what the worker says, and the output data is the warnings and alerts that are the results of the analysis.

[1530] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1532] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1533] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1534] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1535] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1536] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1537] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1538] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1539] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1540] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1541] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1542] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1543] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1544] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1545] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1546] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1547] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1548] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1549] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1550] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1551] The following is further disclosed regarding the above embodiment.

[1552] (Claim 1)

[1553] A means for registering participant information at the start of a conference;

[1554] a means for capturing image data for each participant;

[1555] means for analyzing emotions from the captured image data;

[1556] A means for storing the analyzed emotion data in a database;

[1557] a means for generating real-time feedback based on the emotion data;

[1558] The system includes a means for notifying participants of the generated feedback.

[1559] (Claim 2)

[1560] 2. The system according to claim 1, further comprising means for suggesting countermeasures to the user when the analyzed emotion data indicates doubt or anxiety.

[1561] (Claim 3)

[1562] A means of analyzing the content of participants' comments and detecting whether the comments constitute harassment;

[1563] 10. The system of claim 1, further comprising means for providing warnings and alerts when harassment is detected.

[1564] (Claim 4)

[1565] A means of continually evaluating sentiment data based on the progress of the deal;

[1566] 2. The system according to claim 1, further comprising means for estimating the probability of success of a business negotiation in real time and notifying the user of the probability.

[1567] (Claim 5)

[1568] 10. The system of claim 1, further comprising means for performing a user-selected action in response to the generated feedback.

[1569] "Example 1"

[1570] (Claim 1)

[1571] A means for registering participant information at the start of a conference;

[1572] a means for acquiring image data of each participant;

[1573] A means for analyzing emotions from the acquired image data;

[1574] a means for storing the analyzed emotion data;

[1575] a means for generating real-time feedback based on the emotion data;

[1576] a means for notifying participants of the generated feedback;

[1577] and means for analyzing the speech content of participants and detecting inappropriate remarks.

[1578] (Claim 2)

[1579] 2. The system according to claim 1, further comprising means for suggesting countermeasures to the user when the analyzed emotion data indicates doubt or anxiety.

[1580] (Claim 3)

[1581] 10. The system of claim 1, further comprising means for issuing warnings and alerts when inappropriate speech is detected.

[1582] "Application Example 1"

[1583] (Claim 1)

[1584] A means for registering participant information at the start of a conference;

[1585] a means for capturing image data for each participant;

[1586] means for analyzing emotions from the captured image data;

[1587] A means for storing the analyzed emotion data in a database;

[1588] a means for generating real-time feedback based on the emotion data;

[1589] a means for notifying participants of the generated feedback;

[1590] means for capturing facial image data of a customer;

[1591] means for analyzing customer emotions from the captured facial image data;

[1592] a means for storing customer sentiment data in a database;

[1593] The system includes a means to notify staff in real time of customer responses based on sentiment data.

[1594] (Claim 2)

[1595] 2. The system according to claim 1, further comprising means for suggesting countermeasures to the user when the analyzed emotion data indicates doubt or anxiety.

[1596] (Claim 3)

[1597] 2. The system according to claim 1, further comprising: means for analyzing the content of comments made by participants and detecting whether the comments constitute harassment; and means for issuing a warning and an alert when harassment is detected.

[1598] "Example 2: Combining Emotion Engines"

[1599] (Claim 1)

[1600] A means for registering participant information at the start of a conference;

[1601] a means for capturing image data for each participant;

[1602] means for transmitting the captured image data and audio data to a server;

[1603] means for analyzing emotions from the captured image data and audio data;

[1604] A means for storing the analyzed emotion data in a database;

[1605] a means for generating real-time feedback based on the emotion data;

[1606] a means for notifying participants of the generated feedback;

[1607] A means to continually evaluate and understand the progress of the deal;

[1608] A system that includes a means for analyzing the content of participants' comments and detecting whether the comments constitute harassment.

[1609] (Claim 2)

[1610] 2. The system according to claim 1, further comprising means for suggesting countermeasures to the user when the analyzed emotion data indicates doubt or anxiety.

[1611] (Claim 3)

[1612] 10. The system of claim 1, further comprising means for providing warnings and alerts when harassment is detected.

[1613] "Application example 2 when combining emotion engines"

[1614] (Claim 1)

[1615] A means for registering participant information at the start of a conference;

[1616] a means for capturing image data for each participant;

[1617] means for analyzing emotions from the captured image data;

[1618] A means for storing the analyzed emotion data in a database;

[1619] a means for generating real-time feedback based on the emotion data;

[1620] a means for notifying participants of the generated feedback;

[1621] Analyzing the emotional state of factory workers in their working environment

[1622] a means of providing real-time feedback;

[1623] A system that includes a means to detect signs of harassment during work and send alerts to management.

[1624] (Claim 2)

[1625] 2. The system according to claim 1, further comprising means for suggesting countermeasures to the user when the analyzed emotion data indicates doubt or anxiety.

[1626] (Claim 3)

[1627] A means for analyzing the content of the analyzed worker's comments in real time and detecting whether the comments constitute harassment;

[1628] 10. The system of claim 1, further comprising means for providing warnings and alerts when harassment is detected. [Explanation of symbols]

[1629] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for registering participant information at the start of a conference; a means for capturing image data for each participant; means for analyzing emotions from the captured image data; A means for storing the analyzed emotion data in a database; a means for generating real-time feedback based on the emotion data; and means for notifying participants of the generated feedback.

2. 2. The system according to claim 1, further comprising means for suggesting countermeasures to the user when the analyzed emotion data indicates doubt or anxiety.

3. A means of analyzing the content of participants' comments and detecting whether the comments constitute harassment; 10. The system of claim 1, further comprising means for issuing warnings and alerts when harassment is detected.

4. A means of continually evaluating sentiment data based on the progress of the deal; 2. The system according to claim 1, further comprising means for estimating the probability of success of a business negotiation in real time and notifying the user of the probability.

5. 10. The system of claim 1, further comprising means for performing a user-selected action on the generated feedback.

Images