System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring a participant list and analyzing the speech tendency and role of each participant by referring to a past conference database, a means for analyzing a video feed and a voice feed and performing the face recognition and voice analysis of a speaker, and a means for generating an appropriate interjection or comment according to speech contents and presenting it to a user in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While telework and online learning have rapidly become more common in recent years, online meetings have several drawbacks compared to face-to-face meetings. Specifically, communication can sometimes be hindered by participants finding it difficult to gauge the timing and mood of the conversation. It's also difficult to manage the progress of the agenda, leading to meetings running long or ending without important topics being discussed. Furthermore, managing records and feedback after the meeting is cumbersome, creating the problem of insufficient utilization of information to improve future meetings. [Means for solving the problem]

[0005] In order to solve these problems, the present invention proposes the following means.

[0006] First, it provides a means to obtain a participant list and refer to a database of past meetings to analyze each participant's speaking tendencies and role. This allows for a prior understanding of participant characteristics and promotes appropriate communication. Second, by incorporating means to analyze video and audio feeds and perform facial recognition and audio analysis of speakers, it becomes possible to generate appropriate backchannels and comments based on what is being said and present them to the user in real time. This helps participants maintain a natural conversation rhythm.

[0007] Furthermore, by providing a means to monitor the progress of meetings and send reminders when progress on the agenda is not being made or time is running out, meetings can proceed smoothly and important topics can be addressed appropriately. It also provides a means to record meeting logs, save the content of comments and the progress of agenda items, and generate analytical reports based on this information for use in future meetings. This streamlines post-meeting feedback and record management, enabling continuous improvement.

[0008] A "participant list" is a list containing details such as names and positions of all people participating in an online meeting.

[0009] The "past conference database" is a database that stores logs, minutes, speech records, etc. of past conferences.

[0010] "Speech tendency" refers to the tendency of a particular participant to speak during a meeting, such as how often they speak and what they speak about.

[0011] "Roles" refer to the specific duties and responsibilities that each participant must fulfill in the meeting.

[0012] A "video feed" is a digital signal that provides a real-time image of each participant in a meeting.

[0013] An "audio feed" is a digital signal that provides the real-time audio of each participant in a conference.

[0014] "Facial recognition" is a technology that automatically identifies the face of a specific person from video and processes that information as digital data.

[0015] "Voice analysis" is a technology that processes voice signals as digital data and understands the meaning and intent of the content.

[0016] "Ai-chi" is a short reply or comment inserted at the right time in response to someone else's statement.

[0017] A "comment" is a statement that provides additional opinions or evaluations regarding other people's statements.

[0018] "Reminding" is the act of notifying participants again of important information or important points.

[0019] A "meeting log" is a document or data that records all statements, actions, and progress of agenda items during a meeting.

[0020] An "analysis report" is a document that objectively evaluates and reports on the effectiveness of a meeting and areas for improvement based on collected data. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention provides a system called "Video Chat AI" that facilitates smooth communication during online meetings. This system links multiple functions on the server and terminal sides to efficiently and effectively conduct meetings.

[0043] System Overview

[0044] server:

[0045] The server receives advance notification of the meeting and a list of participants, and refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes the video and audio feeds during the meeting in real time, performing facial recognition and audio analysis of the speakers. It generates backchannels and comments in real time and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[0046] Device:

[0047] The device displays the user's responses and comments received from the server in real time, sends the user's responses and operations to the server, and records the meeting log, providing a saving and editing function so that the user can check it after the meeting has ended.

[0048] A natural language explanation of the program's processing

[0049] 1. User Awareness and Initial Setup:

[0050] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device side, the video chat software launches when the meeting starts, and the user enters their account information and sends authentication information to the server.

[0051] 2. Analyzing video and audio feeds:

[0052] The server analyzes the video and audio feeds in real time, performs facial recognition and speech analysis of the speaker, generates appropriate responses and comments, and sends them to the device, where they are displayed to the user and receive their responses.

[0053] 3. Meeting Management Support:

[0054] The server monitors the progress of the meeting, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which displays it to the user so that the user can smoothly proceed with the meeting.

[0055] 4. Meeting logging and feedback:

[0056] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[0057] Specific examples

[0058] For example, consider the case where User A participates in an online conference. The server analyzes User A's past speaking habits in advance, and generates appropriate backchannels and comments while analyzing User A's comments in real time during the conference. As User A continues to speak, backchannels sent from the server (e.g., "I see, that's interesting") are displayed on the device, smoothly advancing the conversation. Furthermore, as the conference draws to a close, the server generates a reminder saying, "There are 10 minutes left," and the device notifies User A of this, allowing the conference to proceed efficiently.

[0059] This allows users to maintain natural communication during online meetings and ensure important points are covered. Furthermore, after the meeting, users can identify areas for improvement for the next meeting through log recording and feedback, enabling continuous improvement.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] server:

[0063] Receive advance notice of meetings, obtain a list of participants, and analyze each participant's speaking habits and roles by referencing a database of past meetings.

[0064] Device:

[0065] When the meeting begins, the video chat software launches, the user enters their account information, and the authentication information is sent to the server.

[0066] User:

[0067] Users input their speaking preferences and AI behavior settings into the interface.

[0068] Step 2:

[0069] server:

[0070] It analyzes video and audio feeds in real time, performs facial recognition and audio analysis of the speaker, and generates appropriate responses and comments.

[0071] Device:

[0072] Displays the user any backchannels or comments received from the server. Sends the user's responses and actions to the server.

[0073] User:

[0074] The user can review the AI's suggestions and choose to accept or ignore them. The AI ​​responds with feedback and additional information in real time while the user is speaking.

[0075] Step 3:

[0076] server:

[0077] Monitor meeting progress and generate reminders if the agenda is falling behind or if the meeting is running out of time.

[0078] Device:

[0079] Display a reminder message to the user. Send the user's response or operation to the server.

[0080] User:

[0081] The user checks the reminder message and adjusts the progress of the meeting as necessary.

[0082] Step 4:

[0083] server:

[0084] Record a meeting log, saving what was said and what was discussed, and generating an analytical report after the meeting.

[0085] Device:

[0086] After the meeting, the recorded log and generated analysis report are provided to the user, and the user can edit and save the log as needed.

[0087] User:

[0088] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance.

[0089] Step 5:

[0090] server:

[0091] Receive feedback from users and make adjustments to be utilized in future meetings.

[0092] Device:

[0093] Optimize settings for your next meeting based on accumulated data and user feedback.

[0094] User:

[0095] Users can use the saved meeting logs and analysis data to prepare for their next meeting.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] In online meetings, communication between participants can sometimes be disrupted. Participants often lose their ability to speak, or the agenda often doesn't move forward. Furthermore, recording and post-meeting analysis of meeting content is often inadequate, resulting in a lack of feedback to improve future meetings. A solution to these problems is needed.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes a means for acquiring identification information for the conference and analyzing the behavioral tendencies and roles of each participant by referring to past information records, a means for analyzing image and audio feeds to perform face recognition and acoustic analysis of speakers, a means for generating appropriate reactions and comments based on the content of comments and presenting them to participants in real time, a means for monitoring the progress of the conference and issuing warnings if the agenda is not progressing or time is running out, a means for saving the conference record and generating analysis results to utilize the content of comments and the progress of the agenda for future conferences, and a function for transmitting the generated reactions and comments to terminals and presenting them to conference participants in real time. This enables smooth communication in online conferences and efficient recording and analysis of conference content.

[0101] "Conference identification information" is data that identifies a particular conference and includes information about participants and agenda items.

[0102] "Past information records" are data that record the content of statements and progress of previous meetings.

[0103] "Means for analyzing behavioral tendencies and roles" is a function that analyzes the characteristics and roles of each participant based on their past comments and participation status.

[0104] An "image feed" is video data transmitted in real time from a video device such as a camera.

[0105] An "acoustic feed" is audio data transmitted in real time from an audio device such as a microphone.

[0106] "Speaker facial recognition" is a technology that analyzes video data to identify specific people.

[0107] The means of "acoustic analysis" is a function that analyzes audio data to understand the speaker's voice and content.

[0108] "Means for generating reactions and mentions" is a function that automatically creates responses and comments based on the analyzed content of statements.

[0109] "Means for monitoring the progress of a meeting" refers to a function that monitors the progress of agenda items and the allocation of speaking time in real time.

[0110] The "means of issuing a warning" is a function that notifies participants when the meeting is not progressing or when time is running out.

[0111] "Means for saving meeting records" refers to a function that records the contents of a meeting in real time and saves them for later reference.

[0112] The "means for generating analysis results" is a function that analyzes recorded meeting data and provides useful feedback for future meetings.

[0113] "Having the ability to present in real time" refers to technology that allows participants to instantly see their reactions and comments.

[0114] The present invention provides a system for facilitating communication during online conferences. In this system, a server and terminals work in cooperation with each other to support the progress of the conference. Specific embodiments of the system are described below.

[0115] Server Operation

[0116] The server receives advance notice of the meeting, retrieves the list of participants, and analyzes the behavioral trends and roles of each participant by referencing past information records. For this purpose, a database search engine or machine learning model can be used.

[0117] After the meeting starts, the server receives and analyzes the image and audio feeds in real time. It uses facial recognition and acoustic analysis technologies to identify the speaker and analyze the content of their speech. Specifically, OpenCV is used for facial recognition and IBM Watson Speech to Text API is used for audio analysis.

[0118] The server generates appropriate responses and comments based on the content of the comment and sends them to the device in real time using a generative AI model (e.g., GPT-3).

[0119] In addition, the server monitors the progress of the meeting and generates alerts when the agenda is not progressing or time is running out, which are also sent to the terminal in real time and notified to the user.

[0120] After the meeting ends, the server saves the meeting record and generates analysis results based on the content of the comments and the progress of the agenda. These analysis results are provided to the user for use in future meetings.

[0121] Device behavior

[0122] The terminal receives input from the user and sends it to the server. When a conference starts, the terminal launches the video chat software, prompts the user to enter their account information, and sends the authentication information to the server. After authentication, the conference interface is displayed.

[0123] During the meeting, real-time reactions and comments sent from the server are displayed, allowing users to maintain natural conversations. Alerts from the server are also displayed to support the progress of the meeting.

[0124] After the meeting, the device provides the user with a recorded meeting log and recording, which the user can review and edit as necessary.

[0125] Specific examples

[0126] When User A participates in an online conference, the server analyzes User A's past speaking habits in advance and generates appropriate responses and comments. For example, when User A makes a comment during a conference, the server generates a response such as "That's interesting. Please tell us more about it," and sends it to the device. The device displays this to User A in real time, smoothly advancing the conversation.

[0127] As the meeting draws to a close, the server generates a warning such as "10 minutes remaining," and the terminal notifies User A of this, thereby supporting time management.

[0128] This allows User A to maintain natural communication during online meetings and efficiently progress important agenda items. In addition, the logs and feedback recorded after the meeting allow User A to identify areas for improvement in the next meeting.

[0129] Example prompts for generative AI models

[0130] Here are some example prompts you can enter into your generative AI model:

[0131] In an online conferencing system using "video chat AI," the server identifies the speaker in real time during the meeting and generates appropriate responses and comments. Please explain the specific operations at each step of this system, including the user authentication process and the specific processing of real-time analysis during the meeting.

[0132] Using this prompt, the generative AI model can provide detailed explanations and concrete examples.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] The flow of this system's program processing

[0135] Step 1: User authentication

[0136] Step 2: Prepare for the meeting

[0137] Step 3: Real-time analytics during the meeting

[0138] Step 4: Meeting facilitating support

[0139] Step 5: Record the meeting and provide feedback

[0140] Specific explanation of each processing step

[0141] Step 1: User authentication

[0142] Input: User account information (email address, password)

[0143] Specific behavior:

[0144] The server waits for the user to launch the video chat software and enter their account information. When the user enters their email address and password, the device sends this to the server, which then compares the received information with a database of past authentications and authenticates the user.

[0145] Data processing and calculation:

[0146] The server hashes the entered account information and compares it with an existing database. If authentication is successful, the server sends an authentication success message to the terminal.

[0147] Output: Authentication result (success / failure), if successful, the conference interface is displayed

[0148] Step 2: Prepare for the meeting

[0149] Input: Participant list, past conference database

[0150] Specific behavior:

[0151] The server receives advance notice of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze the behavioral trends and roles of each participant. The user checks the start time of the meeting and makes any necessary preparations. The server then sends a reminder before the meeting.

[0152] Data processing and calculation:

[0153] Based on the participant list, past meeting data is extracted from the database, and an analytical algorithm is used to analyze behavioral trends and roles. The analysis results are stored on the server and used for real-time analysis during the meeting.

[0154] Output: Participant behavioral trends and roles, reminder notifications

[0155] Step 3: Real-time analytics during the meeting

[0156] Input: Video feed, audio feed

[0157] Specific behavior:

[0158] When a meeting begins, the server acquires and analyzes video and audio feeds in real time. It performs facial recognition and audio analysis of the speaker, generates appropriate responses and comments, and sends them to the device. The user can view these on the screen. Specifically, it uses OpenCV for facial recognition and IBM Watson Speech to Text API for audio analysis.

[0159] Data processing and calculation:

[0160] Facial recognition algorithms are used to identify speakers from video feeds, and speech recognition technology is used to convert speech into text from audio feeds, which is then used to generate real-time responses and comments using generative AI models (e.g., GPT-3).

[0161] Output: Real-time reactions and comments (sent to terminal)

[0162] Step 4: Meeting facilitating support

[0163] Input: Meeting progress, time management data

[0164] Specific behavior:

[0165] The server monitors the progress of the meeting and generates a warning if the agenda is not progressing or if time is running out. This information is sent to the terminal in real time and notifies the user.

[0166] Data processing and calculation:

[0167] It analyzes progress and time management data in real time and generates alerts based on set conditions, such as when no progress is made on an agenda item for a certain period of time or when a meeting is 10 minutes away from the scheduled end time.

[0168] Output: Warning message (sent to terminal)

[0169] Step 5: Record the meeting and provide feedback

[0170] Input: Meeting video feed, audio feed, speech log

[0171] Specific behavior:

[0172] The server stores the meeting records and generates analysis results based on the content of comments and the progress of the agenda. After the meeting, users can check the logs and recordings and make any necessary edits.

[0173] Data processing and calculation:

[0174] It stores the video and audio feeds of the meeting and analyzes speech logs to generate insights that can be used for future meetings.

[0175] Output: Meeting logs, recordings, and analysis results (provided to users)

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] With conventional online conference systems and virtual store systems, it was difficult to maintain smooth communication with participants and customers. In particular, important topics could not be advanced smoothly during the meeting or customer service process, leaving participants and customers frustrated. This led to a decline in meeting efficiency and customer service quality, ultimately resulting in a deterioration in overall business performance.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for acquiring a participant list and analyzing the speaking tendencies and roles of each participant by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the face and perform audio analysis of the speaker, means for generating appropriate interjections and comments according to the content of the speech and presenting them to the user in real time, and means for analyzing the facial expressions and speech content of customers in real time and providing comments and advice on appropriate ways to respond, thereby enabling the smooth progress of the conference and customer service and smooth communication that satisfies users and customers.

[0181] A "participant list" is a list containing information about people who will be participating in a conference.

[0182] The "past conference database" is a collection of information that stores records and details of conferences that have been held in the past.

[0183] "Speech trends" indicate the characteristics of the content and patterns of speech that a particular participant often makes during a meeting.

[0184] "Roles" refer to the duties and responsibilities of each participant in a meeting.

[0185] A "video feed" is a continuous stream of video data captured through a camera.

[0186] An "audio feed" is a continuous stream of audio data captured through a microphone.

[0187] "Facial recognition" is a technology that analyzes images captured by a camera to identify the face of a specific individual.

[0188] "Voice analysis" is a technology that analyzes the content of speech and the characteristics of the speaker based on acquired voice data.

[0189] "Ai-chi" refers to a reaction or response that shows that you are listening.

[0190] A "comment" is an opinion or additional information about a particular statement.

[0191] "Customer" means a consumer who visits the virtual store to purchase products or services.

[0192] "Facial expressions" are the movements and changes that appear on a person's face to show their emotions.

[0193] "Utterance content" refers to the specific content that is spoken in a conversation or statement.

[0194] "Response methods" refer to the specific means or methods of responding to customers or participants.

[0195] "Comments and Advice" means advice or opinions given based on statements or circumstances.

[0196] "Customer service" refers to the act of explaining and selling products and services to customers.

[0197] A "remind" is a notification or instruction given to remind you of a specific thing.

[0198] "Meeting progress" indicates the progress of the meeting agenda or topic.

[0199] "Logging" means recording the details of what is said and what happens during a meeting or customer service session.

[0200] An "analysis report" is a report summarizing the results of an analysis based on recorded data.

[0201] This invention provides a system that enables smooth communication in virtual stores and online meetings. Specifically, the server and terminals work together to analyze the facial expressions and speech of customers and participants in real time, and provide appropriate responses and advice.

[0202] Hardware and software used

[0203] 1. Hardware:

[0204] Smart glasses (e.g., Google Glass, Microsoft HoloLens)

[0205] Server (e.g. AWS, Google Cloud Platform)

[0206] 2. Software:

[0207] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0208] Facial recognition API (e.g. Microsoft Azure Face API)

[0209] Real-time communication libraries (e.g., WebRTC)

[0210] System Configuration

[0211] The server has the following means:

[0212] 1. A means of obtaining a list of participants and analyzing each participant's speaking tendencies and role by referencing a database of past conferences.

[0213] 2. A means of analyzing the video and audio feeds to perform facial recognition and voice analysis of the speaker.

[0214] 3. A means of generating appropriate responses and comments based on what is being said and presenting them to the user in real time.

[0215] 4. A means of analyzing customers' facial expressions and speech in real time and providing comments and advice on how to respond appropriately.

[0216] The terminal has the following means:

[0217] 1. A means of displaying backchannels and comments received from the server and sending user responses and actions to the server.

[0218] 2. A means of recording past customer service data and providing the ability to review it later.

[0219] Program processing explanation

[0220] The server receives the login information sent by the user from the smart glasses, analyzes the customer's speech patterns and facial expression patterns by referring to past customer service data, analyzes real-time video and audio feeds, and uses facial expression and voice recognition APIs to understand the customer's emotions and speech content. It then generates appropriate responses and advice and sends them to the smart glasses.

[0221] The smart glasses display the responses and advice sent from the server on the device. As the user continues the conversation with the customer, they respond according to the information displayed on the smart glasses. After the conversation is over, the acquired data is sent back to the server and a feedback report is generated for the next time.

[0222] Specific examples

[0223] For example, if staff member A wears smart glasses while serving customers in a virtual store, the server will analyze customer B's purchase history and preferences based on past data. When customer B visits the store, the smart glasses will display advice such as, "Please recommend new products related to products that customer B previously purchased." If customer B asks a question or looks gloomy, the server will analyze that information and provide staff member A with a comment such as, "The customer seems anxious. Please provide a reassuring response."

[0224] Prompt Sentence Examples

[0225] Analysis of customer conversations in a virtual store:

[0226] Perform facial recognition and sentiment analysis on Customer B to generate an appropriate response in real time.

[0227] If Customer B asks, "Can you tell me more about this product?" you respond with, "Of course. The main features of this product are..."

[0228] Store associates use smart glasses to view advice and reminders and help communicate with customers.

[0229] This allows virtual store staff to provide appropriate customer service in real time, improving customer satisfaction.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] The server retrieves the participant list and analyzes each participant's speaking tendencies and roles by referencing a database of past conferences. The server receives the participant list and past conference data as input, performs statistical analysis based on which patterns for each participant are extracted, and generates speaking tendencies data and role information for each participant as output.

[0233] Step 2:

[0234] The server collects video and audio feeds and analyzes them in real time. As input, the video and audio feeds sent from the smart glasses are provided. These feeds are analyzed using facial expression recognition API and speech recognition API to perform facial recognition and speech analysis. As output, analyzed facial expression data and speech content data are obtained.

[0235] Step 3:

[0236] The server generates appropriate backchannels and comments based on the content of the speech. The speech content data obtained in step 2 is used as input. Based on this, appropriate backchannels and comments are generated using natural language processing (NLP) technology. The output is generated as an appropriate response or comment to the user.

[0237] Step 4:

[0238] The server analyzes the customer's facial expressions and speech content in real time, and generates comments and advice indicating the appropriate response method. The facial expression data and speech content data acquired in step 2 are used as input. Based on the analysis results, the server understands the customer's emotional state and requests, and uses a generative AI model to present the appropriate response method. Specific comments and advice are generated as output.

[0239] Step 5:

[0240] The device displays the backchannels, comments, and advice received from the server on the smart glasses. The backchannels, comments, and advice data sent from the server are used as input. This is displayed in real time on the smart glasses' display and provided to the user. The output is the information that the user actually views.

[0241] Step 6:

[0242] The terminal records past customer service data and generates a feedback report for the next time. Input includes user responses and conversations with customers. This data is recorded as a log and sent to the server. The server analyzes the data and compiles it into a feedback report to be used in the next customer service. As output, a feedback report is generated and provided to the user.

[0243] Through these steps, a system will be established that provides appropriate support for customer service in virtual stores in real time, improving customer satisfaction.

[0244] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0245] This invention provides a "Video Chat AI" system for smooth communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions on the server side and the terminal side to efficiently and effectively conduct meetings.

[0246] System Overview

[0247] server:

[0248] The server receives advance notification of the meeting and a list of participants, and then refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes video and audio feeds in real time, performing facial and audio analysis of speakers. It also analyzes the user's emotions using an emotion recognition engine, generates appropriate responses and comments, and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[0249] Device:

[0250] The device displays the user's responses and comments received from the server in real time, and sends the user's responses and actions to the server. It also displays suggestions and measures based on the results of user emotion analysis, and manages the atmosphere of the meeting. It also records the meeting log and provides saving and editing functions so that the user can review it after the meeting has ended.

[0251] A natural language explanation of the program's processing

[0252] 1. User Awareness and Initial Setup:

[0253] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device, the video chat software launches when the meeting begins, and the user enters their account information and sends authentication information to the server. The user then inputs their speaking preferences and AI behavior settings into the interface.

[0254] 2. Analyzing video and audio feeds:

[0255] The server analyzes the video and audio feeds in real time, recognizing the speaker's face and voice. It also uses an emotion recognition engine to analyze the user's emotions and generate appropriate responses and comments. The device displays these responses and comments, along with suggestions and solutions based on the emotion analysis, and receives the user's responses.

[0256] 3. Meeting Management Support:

[0257] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or if time is running out. Furthermore, if the user's emotions change in a negative direction, it generates suggestions and measures to ease the atmosphere. These reminders and suggestions are displayed to the user on the device, and the user's responses and actions are sent to the server.

[0258] 4. Meeting logging and feedback:

[0259] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[0260] Specific examples

[0261] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[0262] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, it is possible to identify areas for improvement for future meetings.

[0263] The processing flow will be explained below.

[0264] Step 1:

[0265] server:

[0266] Receive advance notice of upcoming meetings, obtain a list of participants, and analyze each participant's speaking habits and role by referencing a database of past meetings. This allows you to estimate in advance what each participant will contribute and be prepared to provide appropriate support.

[0267] Device:

[0268] When a meeting starts, the video chat software launches. The user enters their account information and sends the authentication information to the server. This allows the software to provide optimal assistance based on each user's settings and past speaking habits.

[0269] User:

[0270] Users input their speaking preferences and AI behavior settings into the interface, such as "don't interrupt me" and "provide timely advice."

[0271] Step 2:

[0272] server:

[0273] The system analyzes video and audio feeds in real time, recognizes the speaker's face and voice, and then uses an emotion recognition engine to analyze the user's emotions. Based on this, appropriate responses and comments are generated and sent to the device. For example, if User A is nervous while speaking, the generated comments might include something like "Relax."

[0274] Device:

[0275] The system displays the backchannels and comments received from the server to the user. It also displays suggestions and measures based on emotion analysis. The system sends the user's responses and actions to the server, enabling real-time feedback.

[0276] User:

[0277] The user can review the AI's suggestions and choose to accept or ignore them. For example, if the AI ​​suggests "Let's move on to the next agenda item," the user can choose to accept or ignore the suggestion.

[0278] Step 3:

[0279] server:

[0280] Monitor the progress of the meeting. Generate reminders if the agenda is not progressing smoothly or if there is little time left in the meeting. Also, if the user's emotions change to a negative direction, generate suggestions or measures to ease the atmosphere. For example, if it feels like the meeting is dragging on, generate a suggestion such as "Let's take a break."

[0281] Device:

[0282] It displays reminder messages and suggestions to users and sends user responses and actions to the server, ensuring smooth progress of the meeting.

[0283] User:

[0284] The user can check the reminder messages and suggestions and adjust the progress of the meeting as necessary, for example, by following the suggestions to "move on to the next agenda item" or "spend more time on the current agenda item."

[0285] Step 4:

[0286] server:

[0287] Record a meeting log, save what was said and the progress of the agenda, and generate an analysis report after the meeting, which can be used for future meetings.

[0288] Device:

[0289] After the meeting, the recorded logs and generated analysis reports are provided to the user. The user can also edit and save the logs as needed, making record management easier.

[0290] User:

[0291] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance, helping to identify areas for improvement and optimizing the system for future use.

[0292] Step 5:

[0293] server:

[0294] The system receives feedback from users and makes adjustments to be applied to future meetings, thereby enabling continuous improvement of the system.

[0295] Device:

[0296] For your next meeting, we'll optimize your settings based on accumulated data and user feedback, ensuring effective support for future meetings.

[0297] User:

[0298] Users can use the saved meeting logs and analysis data to prepare for their next meeting, thereby improving the efficiency and effectiveness of meetings.

[0299] Example 2

[0300] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0301] In online meetings, it is difficult to properly manage the emotions of speakers and the progress of the meeting in real time. Particularly when there are many participants, it is necessary to grasp the emotions and progress of each speaker and automatically provide appropriate responses and reminders, but conventional systems have not been able to achieve this. Therefore, improving the efficiency of meetings and ensuring smooth communication between participants is a challenge.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0303] In this invention, the server includes means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the speaker's face and perform voice analysis, means for analyzing the user's emotions using an emotion recognition engine, generating appropriate backchannels and comments, and presenting them to the user in real time, means for monitoring the progress of the conference and generating reminders when progress on the agenda is not being made or time is running out, and means for recording a conference log and saving the content of comments and the progress of the agenda. This makes it possible to grasp user emotions and the progress of the conference in real time and respond appropriately.

[0304] A "participant list" is a list that includes the names and IDs of all users who will be participating in a conference.

[0305] A "conference database" is a database that stores information about past conferences, and stores data on what was said and the roles of participants.

[0306] "Speech trends" are behavioral patterns obtained by analyzing the content, frequency, and role of each participant in past meetings.

[0307] "Facial recognition" is a technology that analyzes facial images obtained from a video feed and identifies the speaker's face.

[0308] "Speech analysis" is a technology that analyzes audio feeds and converts what is being said into text.

[0309] An "emotion recognition engine" is an engine that analyzes video and audio data to detect a user's emotional state (e.g., joy, sadness, tension, etc.).

[0310] "Responses and comments" refer to feedback and reactions generated based on what is said, and are elements that help smooth the flow of conversation.

[0311] "Reminders" are notifications and alerts about meeting progress and time management, designed to prevent delays and time-outs.

[0312] A "meeting log" refers to a record of all data such as what was said during the meeting, the progress, and the degree to which the agenda was achieved.

[0313] "Suggestions and measures" are specific actions generated based on emotion analysis to improve the user's condition or the progress of the meeting.

[0314] This invention is a "Video Chat AI" system for facilitating communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions of the server and terminals to efficiently and effectively conduct meetings.

[0315] 1. User Awareness and Initial Setup:

[0316] server:

[0317] The server receives advance notice of the meeting and retrieves the attendee list, for example, using the Google Calendar API to retrieve the meeting schedule.

[0318] The server retrieves relevant data from a past conference database (e.g., MySQL) and analyzes participants' speaking tendencies and roles.

[0319] Device:

[0320] When a user joins a meeting, video chat software (e.g., Zoom, Microsoft Teams) is launched.

[0321] The user enters their account information and sends their authentication information to the server to log in.

[0322] Users input their preferred speaking style and AI behavior settings into the interface.

[0323] 2. Analyzing video and audio feeds:

[0324] server:

[0325] The server analyzes the video and audio feeds in real time, using OpenCV for face recognition of the speaker and the Google Cloud Speech-to-Text API for speech analysis.

[0326] An emotion recognition engine (e.g., IBM Watson Emotion Recognition) is used to analyze the user's emotions and generate appropriate responses and comments.

[0327] Device:

[0328] Display received responses and comments to the user.

[0329] Suggestions and measures based on sentiment analysis are displayed to the user.

[0330] 3. Meeting Management Support:

[0331] server:

[0332] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or time is running out.

[0333] Device:

[0334] Display reminder messages and suggestions to the user and send the user's responses and actions to the server.

[0335] 4. Meeting logging and feedback:

[0336] server:

[0337] The server records the meeting log and stores the content of the discussion and the progress of the agenda in a database.

[0338] After the meeting, the generated logs and analysis reports are provided to the user so that they can be used in future meetings.

[0339] Device:

[0340] Users can review meeting recordings and logs and edit them as needed.

[0341] Users can provide feedback and rate the AI's performance to identify areas for improvement in future meetings.

[0342] Specific examples

[0343] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[0344] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, improvements can be made to future meetings.

[0345] Prompt Sentence Examples

[0346] Here are some example prompts to input to the generative AI model:

[0347] I have a question about the functionality of the "Video Chat AI" system I will be using in my next online meeting. Could you please explain in detail how this system, which is equipped with an emotion recognition engine, analyzes the user's emotions and generates appropriate responses and comments? Also, could you please explain in detail how it will support the progress of the meeting?

[0348] This system ensures smooth progress of meetings and manages participants' emotions.

[0349] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0350] Step 1: Prepare for the meeting

[0351] server:

[0352] The server receives advance notices of upcoming meetings. As input, meeting schedule information is provided, e.g., data retrieved from the Google Calendar API.

[0353] The server generates a participant list based on this input data and retrieves data related to each participant from a past conference database (e.g., MySQL). The output is a participant list and the results of an analysis of past speaking tendencies and roles.

[0354] Device:

[0355] The device will send the generated participant list and meeting notification to the user via email or in-app notification.

[0356] Specific behavior:

[0357] The attendee list and meeting materials are displayed on the screen for easy user confirmation, and meeting notifications are sent via email and in-app notifications.

[0358] Step 2: Start the meeting and authenticate users

[0359] server:

[0360] The server receives a signal to start a conference, and inputs include a conference start instruction and each user's authentication information.

[0361] The server connects to a SaaS (Software as a Service) service (e.g., Zoom, Microsoft Teams) to start the conference. The output is the conference room URL and the user authentication result.

[0362] Device:

[0363] The terminal starts the video chat software and sends the user's account information to the server, with the input data being the user's authentication information.

[0364] If the authentication information is successful, the terminal displays the conference room URL returned by the server. The output is the result of providing the conference room URL and verifying the settings information.

[0365] User:

[0366] Users receive a URL or code for joining the meeting and join the meeting.

[0367] The user inputs their preferred speech style and AI behavior settings into the interface. The input data is user preference information, which is then sent to the server.

[0368] Specific behavior:

[0369] The server processes the authentication information and, if successful, provides the device with a meeting room URL, which the user can use to join the meeting in real time.

[0370] Step 3: Analyze the video and audio feeds

[0371] server:

[0372] The server analyzes the video and audio feeds in real time. The input data are the video and audio feeds sent from the devices.

[0373] The speaker's face is recognized using OpenCV, and speech analysis is performed using the Google Cloud Speech-to-Text API. Emotions are analyzed using an emotion recognition engine (e.g., IBM Watson Emotion Recognition). The output is the speaker's face position, speech analysis results, and emotion analysis results.

[0374] Device:

[0375] Displays the backchannels and comments received from the server. The input data is the analysis results sent from the server. The output is the display of feedback to the user.

[0376] Specific behavior:

[0377] When the user starts speaking, the device sends video and audio data to the server, which returns the analysis results to the device and displays them to the user.

[0378] Step 4: Sentiment analysis and response

[0379] server:

[0380] An emotion recognition engine is used to analyze the user's emotions, and the input data is video and audio feed from the device.

[0381] Based on the results of the sentiment analysis, appropriate responses and comments are generated and sent to the device. The output is the sentiment analysis results and the comments and responses based on them.

[0382] Device:

[0383] The received comments and responses are displayed to the user. The input data is the emotion analysis results sent from the server. The output is the content displayed to the user.

[0384] Specific behavior:

[0385] If the user is nervous, display a pop-up notification such as "Would you like to take a short break?"

[0386] Step 5: Managing the meeting

[0387] server:

[0388] The server monitors the progress of the conference, and input data includes the progress status and time information of the conference.

[0389] If the agenda is not progressing or time is running out, a reminder is generated and sent to the terminal. The output is the reminder message.

[0390] Device:

[0391] Display reminders received from the server to the user. The input data is the reminder message sent from the server. The output is the notification content to the user.

[0392] Specific behavior:

[0393] When there is little time left, a notification such as "10 minutes remaining" will be displayed.

[0394] Step 6: Meeting logging and feedback

[0395] server:

[0396] The server records the conference log, and input data includes the content of the conference and its progress.

[0397] The system stores the content of comments and the progress of the agenda in a database, and provides the user with logs and analysis reports for use in future meetings. The output is a meeting log and analysis report.

[0398] Device:

[0399] The conference log and analysis report received from the server are displayed so that the user can check them. The input data are the conference log and analysis report. The output is the content displayed to the user.

[0400] It provides an interface for users to input feedback and evaluate the AI's performance. The input is the user's feedback. The output is the feedback result.

[0401] Specific behavior:

[0402] After the meeting, the recording and log of the meeting are provided to the user, who can review them and provide feedback. This will help evaluate the AI's performance and help improve it for future meetings.

[0403] (Application example 2)

[0404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] Analyzing employee conversations and emotions in real time and providing appropriate feedback during factory meetings and conferences is a difficult task. It is particularly important to ensure that meetings proceed smoothly and create an environment where employees can participate without feeling nervous or stressed. It is also necessary to efficiently manage meeting logs and recordings and analyze the information for use in future meetings.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0407] In this invention, the server includes: means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referencing a past conference database; means for analyzing video and audio feeds to recognize the speaker's face and perform voice analysis; means for generating appropriate backchannels and comments based on the content of the speech and presenting them to the user in real time; and means for analyzing user emotions in real time using an emotion recognition engine and generating proposals and countermeasures based on the emotion analysis results. This enables real-time and appropriate feedback and emotion management in factory conferences and meetings. It also enables smooth progress in meetings and provides an environment where employees can participate with peace of mind. Furthermore, the recorded meeting logs and recordings can be used to improve future meetings.

[0408] A "participant list" is a list of people participating in a conference or meeting, and includes information for understanding the role and past speaking tendencies of each participant.

[0409] A "conference database" is a collection of information that stores data such as logs, audio recordings, and video recordings of past meetings, and is used to analyze the speaking tendencies and roles of each participant.

[0410] "Speech tendency" refers to a pattern that indicates what each participant talks about and how often they speak in a conference or meeting.

[0411] "Roles" indicate the position and scope of responsibility of each participant in a conference or meeting, and include, for example, leader, administrative staff, technical staff, etc.

[0412] A "video feed" is a stream of video data that broadcasts video of conference or meeting participants in real time.

[0413] An "audio feed" is a stream of audio data that delivers the voices of conference or meeting participants in real time.

[0414] "Facial recognition" is a technology that analyzes video data to identify specific people, and is used to identify speakers.

[0415] "Voice analysis" is a technology that analyzes voice data to understand what is being said, and is used to generate appropriate responses and comments.

[0416] An "emotion recognition engine" is software or hardware for analyzing a user's emotional state from facial expressions, voice, etc.

[0417] "Ai-chi" is a short phrase or expression used during a conversation to show understanding and empathy for what the other person is saying, and is used to facilitate communication.

[0418] A "comment" is a word used to express an appropriate opinion or reaction to something said during a conference or meeting.

[0419] A "proposal" is a specific idea or action point that encourages new opinions or actions based on what was said during a conference or meeting.

[0420] "Countermeasures" are specific actions or methods proposed to solve the problem based on the content of the comments and the results of emotion analysis.

[0421] A "log" is data that records what is said during a conference or meeting, the progress of the agenda, etc.

[0422] "Recording" refers to data that records video and audio of a conference or meeting.

[0423] An "analysis report" is a document that evaluates the effectiveness and progress of a meeting based on recorded logs and video data, and summarizes the analysis results to be used in future meetings.

[0424] "Reminds" are notifications and alerts that monitor the progress of meetings and conferences and prompt necessary action if things are not going as planned or time is running out.

[0425] This invention provides a "communication support robot system" for smoothly conducting conferences and meetings within a factory. How the system is implemented will be explained below in detail.

[0426] System Overview

[0427] This system mainly consists of a server, robot terminals, and users. The server is responsible for storing and analyzing meeting data and for real-time communication. The robot terminals are devices that allow direct interaction with users within the factory and are equipped with sensors such as cameras and microphones.

[0428] Server Features

[0429] 1. Participant list acquisition and conference database reference function:

[0430] The server receives advance notice of the meeting and a list of participants, and analyzes each participant's speaking tendencies and roles by referring to a database of past meetings.

[0431] 2. Video and audio feed analysis:

[0432] The server analyzes the video and audio feeds sent from the robot terminal in real time, performs facial recognition and voice analysis of the speaker, and analyzes the user's emotions using an emotion recognition engine (e.g., Microsoft's Azure Cognitive Services).

[0433] 3. Sentiment analysis-based suggestion and countermeasure generation function:

[0434] The server generates appropriate responses and comments based on the analysis results, and if necessary, suggests ways to relax. This data is then sent to the robot terminal in real time.

[0435] 4. Meeting progress support function:

[0436] The server monitors the progress of the meeting and generates reminders if the meeting is not progressing as planned or if time is running out.

[0437] 5. Meeting logging and analysis report generation function:

[0438] The server records the meeting log, saving what was said and the progress of the agenda, and generates an analytical report for later review.

[0439] Robot terminal functions

[0440] 1. User input and authentication:

[0441] When the meeting starts, the robot terminal launches the video chat software, the user enters account information into the interface, and the authentication information is sent to the server.

[0442] 2. Get real-time feeds:

[0443] The robot terminal uses a camera and microphone to capture the user's video and audio feeds in real time and transmits them to a server.

[0444] 3. Display of suggestions and measures:

[0445] The system displays to the user the responses, comments, and suggestions and measures based on sentiment analysis sent from the server, and receives the user's responses.

[0446] 4. Processing example:

[0447] For example, if a leader is detected to be nervous during a presentation at a quality control meeting in a factory, the robot will respond by saying, "It's okay, stay calm. What I've just explained is very important."

[0448] The robot will also detect if the agenda is not progressing during a meeting and display a reminder, saying, "We need to confirm the next step on this topic."

[0449] Prompt Sentence Examples

[0450] "If you sense tension in a meeting, a prompt will suggest:

[0451] Generate suggestions to ease the leader's tension by saying, "It's okay to stay calm. What I just explained is very important."

[0452] In this way, the communication support robot system can smoothly conduct conferences and meetings within factories and provide real-time support for employee emotional management, thereby improving meeting efficiency and reducing stress for participants.

[0453] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0454] Step 1:

[0455] Obtaining and pre-analyzing the participant list

[0456] Before the meeting starts, the server receives advance notice of the meeting, obtains the participant list, and then analyzes each participant's speaking habits and roles by referring to a database of past meetings.

[0457] Input: Meeting advance notices, participant list, past meeting database

[0458] Data processing and calculation: The server reads the conference database and compares it with the participant list, analyzing the frequency of each participant's comments and topic trends.

[0459] Output: Analysis results (commentary trends, roles)

[0460] Step 2:

[0461] Conference start and user authentication

[0462] When the conference starts, the terminal launches the video chat software, prompts the user to enter account information, and transmits the authentication information to the server.

[0463] Input: User account information, authentication information

[0464] Data processing and data calculation: The terminal sends the entered account information to the server, and the server verifies the authentication information and confirms the user.

[0465] Output: Authentication result (authentication success / failure)

[0466] Step 3:

[0467] Acquiring and sending video and audio feeds

[0468] The device uses a camera and microphone to capture the user's video and audio feeds in real time during the meeting and transmits them to the server.

[0469] Input: Video feed, Audio feed

[0470] Data processing and data calculation: The device sends the acquired video and audio feeds to the server.

[0471] Output: Video feed and audio feed sent

[0472] Step 4:

[0473] Real-time video and audio analysis

[0474] The server analyzes the received video and audio feeds in real time, performs face recognition and voice analysis of the speaker, and also analyzes the user's emotions using an emotion recognition engine.

[0475] Input: Video feed, Audio feed

[0476] Data processing and data calculation: The server uses a facial recognition algorithm and a voice analysis engine to identify the speaker, and an emotion recognition engine to analyze the user's emotions.

[0477] Output: Face recognition results, voice analysis results, emotion analysis results

[0478] Step 5:

[0479] Generate and present backchannels and comments

[0480] The server generates appropriate responses and comments based on the content of the speech and the results of emotion analysis, and sends them to the device, which then displays them to the user and outputs them as audio.

[0481] Input: Face recognition results, voice analysis results, emotion analysis results

[0482] Data processing and data calculation: The server uses the generative AI model to generate appropriate responses and comments, and sends them to the device.

[0483] Output: backchannel, comments

[0484] Step 6:

[0485] Meeting progress monitoring and reminder generation

[0486] The server monitors the progress of the meeting in real time, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which then notifies the user.

[0487] Input: Meeting progress (log, timestamp)

[0488] Data processing and data calculation: The server analyzes the progress, generates reminders and sends them to the device.

[0489] Output: Reminder notification

[0490] Step 7:

[0491] Recommendation generation based on emotion changes

[0492] If the user's emotional state changes to a negative state, the server generates relaxation suggestions and sends them to the device, which displays them to the user and also outputs them audibly.

[0493] Input: Sentiment analysis results

[0494] Data processing and data calculation: The server uses the generative AI model to generate relaxation suggestions and send them to the device.

[0495] Output: Proposal notification

[0496] Step 8:

[0497] Meeting logging and analytical report generation

[0498] The server records the meeting log, saving the content of comments and the progress of the agenda. After the meeting, an analysis report is generated based on this data, allowing users to use the results in future meetings.

[0499] Input: Meeting log (contents, progress, timestamp)

[0500] Data processing and data calculation: The server stores the log data and generates an analysis report after completion.

[0501] Output: Log data, analysis report

[0502] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0503] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0504] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0505] [Second embodiment]

[0506] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0507] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0508] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0509] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0510] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0511] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0512] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0513] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0514] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0515] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0516] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0517] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0518] This invention provides a system called "Video Chat AI" that facilitates smooth communication during online meetings. This system links multiple functions on the server and terminal sides to efficiently and effectively conduct meetings.

[0519] System Overview

[0520] server:

[0521] The server receives advance notification of the meeting and a list of participants, and refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes the video and audio feeds during the meeting in real time, performing facial recognition and audio analysis of the speakers. It generates backchannels and comments in real time and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[0522] Device:

[0523] The device displays the user's responses and comments received from the server in real time, sends the user's responses and operations to the server, and records the meeting log, providing a saving and editing function so that the user can check it after the meeting has ended.

[0524] A natural language explanation of the program's processing

[0525] 1. User Awareness and Initial Setup:

[0526] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device side, the video chat software launches when the meeting starts, and the user enters their account information and sends authentication information to the server.

[0527] 2. Analyzing video and audio feeds:

[0528] The server analyzes the video and audio feeds in real time, performs facial recognition and speech analysis of the speaker, generates appropriate responses and comments, and sends them to the device, where they are displayed to the user and receive their responses.

[0529] 3. Meeting Management Support:

[0530] The server monitors the progress of the meeting, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which displays it to the user so that the user can smoothly proceed with the meeting.

[0531] 4. Meeting logging and feedback:

[0532] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[0533] Specific examples

[0534] For example, consider the case where User A participates in an online conference. The server analyzes User A's past speaking habits in advance, and generates appropriate backchannels and comments while analyzing User A's comments in real time during the conference. As User A continues to speak, backchannels sent from the server (e.g., "I see, that's interesting") are displayed on the device, smoothly advancing the conversation. Furthermore, as the conference draws to a close, the server generates a reminder saying, "There are 10 minutes left," and the device notifies User A of this, allowing the conference to proceed efficiently.

[0535] This allows users to maintain natural communication during online meetings and ensure important points are covered. Furthermore, after the meeting, users can identify areas for improvement for the next meeting through log recording and feedback, enabling continuous improvement.

[0536] The processing flow will be explained below.

[0537] Step 1:

[0538] server:

[0539] Receive advance notice of meetings, obtain a list of participants, and analyze each participant's speaking habits and roles by referencing a database of past meetings.

[0540] Device:

[0541] When the meeting begins, the video chat software launches, the user enters their account information, and the authentication information is sent to the server.

[0542] User:

[0543] Users input their speaking preferences and AI behavior settings into the interface.

[0544] Step 2:

[0545] server:

[0546] It analyzes video and audio feeds in real time, performs facial recognition and audio analysis of the speaker, and generates appropriate responses and comments.

[0547] Device:

[0548] Displays the user any backchannels or comments received from the server. Sends the user's responses and actions to the server.

[0549] User:

[0550] The user can review the AI's suggestions and choose to accept or ignore them. The AI ​​responds with feedback and additional information in real time while the user is speaking.

[0551] Step 3:

[0552] server:

[0553] Monitor meeting progress and generate reminders if the agenda is falling behind or if the meeting is running out of time.

[0554] Device:

[0555] Display a reminder message to the user. Send the user's response or operation to the server.

[0556] User:

[0557] The user checks the reminder message and adjusts the progress of the meeting as necessary.

[0558] Step 4:

[0559] server:

[0560] Record a meeting log, saving what was said and what was discussed, and generating an analytical report after the meeting.

[0561] Device:

[0562] After the meeting, the recorded log and generated analysis report are provided to the user, and the user can edit and save the log as needed.

[0563] User:

[0564] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance.

[0565] Step 5:

[0566] server:

[0567] Receive feedback from users and make adjustments to be utilized in future meetings.

[0568] Device:

[0569] Optimize settings for your next meeting based on accumulated data and user feedback.

[0570] User:

[0571] Users can use the saved meeting logs and analysis data to prepare for their next meeting.

[0572] Example 1

[0573] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] In online meetings, communication between participants can sometimes be disrupted. Participants often lose their ability to speak, or the agenda often doesn't move forward. Furthermore, recording and post-meeting analysis of meeting content is often inadequate, resulting in a lack of feedback to improve future meetings. A solution to these problems is needed.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0576] In this invention, the server includes a means for acquiring identification information for the conference and analyzing the behavioral tendencies and roles of each participant by referring to past information records, a means for analyzing image and audio feeds to perform face recognition and acoustic analysis of speakers, a means for generating appropriate reactions and comments based on the content of comments and presenting them to participants in real time, a means for monitoring the progress of the conference and issuing warnings if the agenda is not progressing or time is running out, a means for saving the conference record and generating analysis results to utilize the content of comments and the progress of the agenda for future conferences, and a function for transmitting the generated reactions and comments to terminals and presenting them to conference participants in real time. This enables smooth communication in online conferences and efficient recording and analysis of conference content.

[0577] "Conference identification information" is data that identifies a particular conference and includes information about participants and agenda items.

[0578] "Past information records" are data that record the content of statements and progress of previous meetings.

[0579] "Means for analyzing behavioral tendencies and roles" is a function that analyzes the characteristics and roles of each participant based on their past comments and participation status.

[0580] An "image feed" is video data transmitted in real time from a video device such as a camera.

[0581] An "acoustic feed" is audio data transmitted in real time from an audio device such as a microphone.

[0582] "Speaker facial recognition" is a technology that analyzes video data to identify specific people.

[0583] The means of "acoustic analysis" is a function that analyzes audio data to understand the speaker's voice and content.

[0584] "Means for generating reactions and mentions" is a function that automatically creates responses and comments based on the analyzed content of statements.

[0585] "Means for monitoring the progress of a meeting" refers to a function that monitors the progress of agenda items and the allocation of speaking time in real time.

[0586] The "means of issuing a warning" is a function that notifies participants when the meeting is not progressing or when time is running out.

[0587] "Means for saving meeting records" refers to a function that records the contents of a meeting in real time and saves them for later reference.

[0588] The "means for generating analysis results" is a function that analyzes recorded meeting data and provides useful feedback for future meetings.

[0589] "Having the ability to present in real time" refers to technology that allows participants to instantly see their reactions and comments.

[0590] The present invention provides a system for facilitating communication during online conferences. In this system, a server and terminals work in cooperation with each other to support the progress of the conference. Specific embodiments of the system are described below.

[0591] Server Operation

[0592] The server receives advance notice of the meeting, retrieves the list of participants, and analyzes the behavioral trends and roles of each participant by referencing past information records. For this purpose, a database search engine or machine learning model can be used.

[0593] After the meeting starts, the server receives and analyzes the image and audio feeds in real time. It uses facial recognition and acoustic analysis technologies to identify the speaker and analyze the content of their speech. Specifically, OpenCV is used for facial recognition and IBM Watson Speech to Text API is used for audio analysis.

[0594] The server generates appropriate responses and comments based on the content of the comment and sends them to the device in real time using a generative AI model (e.g., GPT-3).

[0595] In addition, the server monitors the progress of the meeting and generates alerts when the agenda is not progressing or time is running out, which are also sent to the terminal in real time and notified to the user.

[0596] After the meeting ends, the server saves the meeting record and generates analysis results based on the content of the comments and the progress of the agenda. These analysis results are provided to the user for use in future meetings.

[0597] Device behavior

[0598] The terminal receives input from the user and sends it to the server. When a conference starts, the terminal launches the video chat software, prompts the user to enter their account information, and sends the authentication information to the server. After authentication, the conference interface is displayed.

[0599] During the meeting, real-time reactions and comments sent from the server are displayed, allowing users to maintain natural conversations. Alerts from the server are also displayed to support the progress of the meeting.

[0600] After the meeting, the device provides the user with a recorded meeting log and recording, which the user can review and edit as necessary.

[0601] Specific examples

[0602] When User A participates in an online conference, the server analyzes User A's past speaking habits in advance and generates appropriate responses and comments. For example, when User A makes a comment during a conference, the server generates a response such as "That's interesting. Please tell us more about it," and sends it to the device. The device displays this to User A in real time, smoothly advancing the conversation.

[0603] As the meeting draws to a close, the server generates a warning such as "10 minutes remaining," and the terminal notifies User A of this, thereby supporting time management.

[0604] This allows User A to maintain natural communication during online meetings and efficiently progress important agenda items. In addition, the logs and feedback recorded after the meeting allow User A to identify areas for improvement in the next meeting.

[0605] Example prompts for generative AI models

[0606] Here are some example prompts you can enter into your generative AI model:

[0607] In an online conferencing system using "video chat AI," the server identifies the speaker in real time during the meeting and generates appropriate responses and comments. Please explain the specific operations at each step of this system, including the user authentication process and the specific processing of real-time analysis during the meeting.

[0608] Using this prompt, the generative AI model can provide detailed explanations and concrete examples.

[0609] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0610] The flow of this system's program processing

[0611] Step 1: User authentication

[0612] Step 2: Prepare for the meeting

[0613] Step 3: Real-time analytics during the meeting

[0614] Step 4: Meeting facilitating support

[0615] Step 5: Record the meeting and provide feedback

[0616] Specific explanation of each processing step

[0617] Step 1: User authentication

[0618] Input: User account information (email address, password)

[0619] Specific behavior:

[0620] The server waits for the user to launch the video chat software and enter their account information. When the user enters their email address and password, the device sends this to the server, which then compares the received information with a database of past authentications and authenticates the user.

[0621] Data processing and calculation:

[0622] The server hashes the entered account information and compares it with an existing database. If authentication is successful, the server sends an authentication success message to the terminal.

[0623] Output: Authentication result (success / failure), if successful, the conference interface is displayed

[0624] Step 2: Prepare for the meeting

[0625] Input: Participant list, past conference database

[0626] Specific behavior:

[0627] The server receives advance notice of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze the behavioral trends and roles of each participant. The user checks the start time of the meeting and makes any necessary preparations. The server then sends a reminder before the meeting.

[0628] Data processing and calculation:

[0629] Based on the participant list, past meeting data is extracted from the database, and an analytical algorithm is used to analyze behavioral trends and roles. The analysis results are stored on the server and used for real-time analysis during the meeting.

[0630] Output: Participant behavioral trends and roles, reminder notifications

[0631] Step 3: Real-time analytics during the meeting

[0632] Input: Video feed, audio feed

[0633] Specific behavior:

[0634] When a meeting begins, the server acquires and analyzes video and audio feeds in real time. It performs facial recognition and audio analysis of the speaker, generates appropriate responses and comments, and sends them to the device. The user can view these on the screen. Specifically, it uses OpenCV for facial recognition and IBM Watson Speech to Text API for audio analysis.

[0635] Data processing and calculation:

[0636] Facial recognition algorithms are used to identify speakers from video feeds, and speech recognition technology is used to convert speech into text from audio feeds, which is then used to generate real-time responses and comments using generative AI models (e.g., GPT-3).

[0637] Output: Real-time reactions and comments (sent to terminal)

[0638] Step 4: Meeting facilitating support

[0639] Input: Meeting progress, time management data

[0640] Specific behavior:

[0641] The server monitors the progress of the meeting and generates a warning if the agenda is not progressing or if time is running out. This information is sent to the terminal in real time and notifies the user.

[0642] Data processing and calculation:

[0643] It analyzes progress and time management data in real time and generates alerts based on set conditions, such as when no progress is made on an agenda item for a certain period of time or when a meeting is 10 minutes away from the scheduled end time.

[0644] Output: Warning message (sent to terminal)

[0645] Step 5: Record the meeting and provide feedback

[0646] Input: Meeting video feed, audio feed, speech log

[0647] Specific behavior:

[0648] The server stores the meeting records and generates analysis results based on the content of comments and the progress of the agenda. After the meeting, users can check the logs and recordings and make any necessary edits.

[0649] Data processing and calculation:

[0650] It stores the video and audio feeds of the meeting and analyzes speech logs to generate insights that can be used for future meetings.

[0651] Output: Meeting logs, recordings, and analysis results (provided to users)

[0652] (Application example 1)

[0653] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0654] With conventional online conference systems and virtual store systems, it was difficult to maintain smooth communication with participants and customers. In particular, important topics could not be advanced smoothly during the meeting or customer service process, leaving participants and customers frustrated. This led to a decline in meeting efficiency and customer service quality, ultimately resulting in a deterioration in overall business performance.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0656] In this invention, the server includes means for acquiring a participant list and analyzing the speaking tendencies and roles of each participant by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the face and perform audio analysis of the speaker, means for generating appropriate interjections and comments according to the content of the speech and presenting them to the user in real time, and means for analyzing the facial expressions and speech content of customers in real time and providing comments and advice on appropriate ways to respond, thereby enabling the smooth progress of the conference and customer service and smooth communication that satisfies users and customers.

[0657] A "participant list" is a list containing information about people who will be participating in a conference.

[0658] The "past conference database" is a collection of information that stores records and details of conferences that have been held in the past.

[0659] "Speech trends" indicate the characteristics of the content and patterns of speech that a particular participant often makes during a meeting.

[0660] "Roles" refer to the duties and responsibilities of each participant in a meeting.

[0661] A "video feed" is a continuous stream of video data captured through a camera.

[0662] An "audio feed" is a continuous stream of audio data captured through a microphone.

[0663] "Facial recognition" is a technology that analyzes images captured by a camera to identify the face of a specific individual.

[0664] "Voice analysis" is a technology that analyzes the content of speech and the characteristics of the speaker based on acquired voice data.

[0665] "Ai-chi" refers to a reaction or response that shows that you are listening.

[0666] A "comment" is an opinion or additional information about a particular statement.

[0667] "Customer" means a consumer who visits the virtual store to purchase products or services.

[0668] "Facial expressions" are the movements and changes that appear on a person's face to show their emotions.

[0669] "Utterance content" refers to the specific content that is spoken in a conversation or statement.

[0670] "Response methods" refer to the specific means or methods of responding to customers or participants.

[0671] "Comments and Advice" means advice or opinions given based on statements or circumstances.

[0672] "Customer service" refers to the act of explaining and selling products and services to customers.

[0673] A "remind" is a notification or instruction given to remind you of a specific thing.

[0674] "Meeting progress" indicates the progress of the meeting agenda or topic.

[0675] "Logging" means recording the details of what is said and what happens during a meeting or customer service session.

[0676] An "analysis report" is a report summarizing the results of an analysis based on recorded data.

[0677] This invention provides a system that enables smooth communication in virtual stores and online meetings. Specifically, the server and terminals work together to analyze the facial expressions and speech of customers and participants in real time, and provide appropriate responses and advice.

[0678] Hardware and software used

[0679] 1. Hardware:

[0680] Smart glasses (e.g., Google Glass, Microsoft HoloLens)

[0681] Server (e.g. AWS, Google Cloud Platform)

[0682] 2. Software:

[0683] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0684] Facial recognition API (e.g. Microsoft Azure Face API)

[0685] Real-time communication libraries (e.g., WebRTC)

[0686] System Configuration

[0687] The server has the following means:

[0688] 1. A means of obtaining a list of participants and analyzing each participant's speaking tendencies and role by referencing a database of past conferences.

[0689] 2. A means of analyzing the video and audio feeds to perform facial recognition and voice analysis of the speaker.

[0690] 3. A means of generating appropriate responses and comments based on what is being said and presenting them to the user in real time.

[0691] 4. A means of analyzing customers' facial expressions and speech in real time and providing comments and advice on how to respond appropriately.

[0692] The terminal has the following means:

[0693] 1. A means of displaying backchannels and comments received from the server and sending user responses and actions to the server.

[0694] 2. A means of recording past customer service data and providing the ability to review it later.

[0695] Program processing explanation

[0696] The server receives the login information sent by the user from the smart glasses, analyzes the customer's speech patterns and facial expression patterns by referring to past customer service data, analyzes real-time video and audio feeds, and uses facial expression and voice recognition APIs to understand the customer's emotions and speech content. It then generates appropriate responses and advice and sends them to the smart glasses.

[0697] The smart glasses display the responses and advice sent from the server on the device. As the user continues the conversation with the customer, they respond according to the information displayed on the smart glasses. After the conversation is over, the acquired data is sent back to the server and a feedback report is generated for the next time.

[0698] Specific examples

[0699] For example, if staff member A wears smart glasses while serving customers in a virtual store, the server will analyze customer B's purchase history and preferences based on past data. When customer B visits the store, the smart glasses will display advice such as, "Please recommend new products related to products that customer B previously purchased." If customer B asks a question or looks gloomy, the server will analyze that information and provide staff member A with a comment such as, "The customer seems anxious. Please provide a reassuring response."

[0700] Prompt Sentence Examples

[0701] Analysis of customer conversations in a virtual store:

[0702] Perform facial recognition and sentiment analysis on Customer B to generate an appropriate response in real time.

[0703] If Customer B asks, "Can you tell me more about this product?" you respond with, "Of course. The main features of this product are..."

[0704] Store associates use smart glasses to view advice and reminders and help communicate with customers.

[0705] This allows virtual store staff to provide appropriate customer service in real time, improving customer satisfaction.

[0706] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0707] Step 1:

[0708] The server retrieves the participant list and analyzes each participant's speaking tendencies and roles by referencing a database of past conferences. The server receives the participant list and past conference data as input, performs statistical analysis based on which patterns for each participant are extracted, and generates speaking tendencies data and role information for each participant as output.

[0709] Step 2:

[0710] The server collects video and audio feeds and analyzes them in real time. As input, the video and audio feeds sent from the smart glasses are provided. These feeds are analyzed using facial expression recognition API and speech recognition API to perform facial recognition and speech analysis. As output, analyzed facial expression data and speech content data are obtained.

[0711] Step 3:

[0712] The server generates appropriate backchannels and comments based on the content of the speech. The speech content data obtained in step 2 is used as input. Based on this, appropriate backchannels and comments are generated using natural language processing (NLP) technology. The output is generated as an appropriate response or comment to the user.

[0713] Step 4:

[0714] The server analyzes the customer's facial expressions and speech content in real time, and generates comments and advice indicating the appropriate response method. The facial expression data and speech content data acquired in step 2 are used as input. Based on the analysis results, the server understands the customer's emotional state and requests, and uses a generative AI model to present the appropriate response method. Specific comments and advice are generated as output.

[0715] Step 5:

[0716] The device displays the backchannels, comments, and advice received from the server on the smart glasses. The backchannels, comments, and advice data sent from the server are used as input. This is displayed in real time on the smart glasses' display and provided to the user. The output is the information that the user actually views.

[0717] Step 6:

[0718] The terminal records past customer service data and generates a feedback report for the next time. Input includes user responses and conversations with customers. This data is recorded as a log and sent to the server. The server analyzes the data and compiles it into a feedback report to be used in the next customer service. As output, a feedback report is generated and provided to the user.

[0719] Through these steps, a system will be established that provides appropriate support for customer service in virtual stores in real time, improving customer satisfaction.

[0720] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0721] This invention provides a "Video Chat AI" system for smooth communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions on the server side and the terminal side to efficiently and effectively conduct meetings.

[0722] System Overview

[0723] server:

[0724] The server receives advance notification of the meeting and a list of participants, and then refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes video and audio feeds in real time, performing facial and audio analysis of speakers. It also analyzes the user's emotions using an emotion recognition engine, generates appropriate responses and comments, and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[0725] Device:

[0726] The device displays the user's responses and comments received from the server in real time, and sends the user's responses and actions to the server. It also displays suggestions and measures based on the results of user emotion analysis, and manages the atmosphere of the meeting. It also records the meeting log and provides saving and editing functions so that the user can review it after the meeting has ended.

[0727] A natural language explanation of the program's processing

[0728] 1. User Awareness and Initial Setup:

[0729] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device, the video chat software launches when the meeting begins, and the user enters their account information and sends authentication information to the server. The user then inputs their speaking preferences and AI behavior settings into the interface.

[0730] 2. Analyzing video and audio feeds:

[0731] The server analyzes the video and audio feeds in real time, recognizing the speaker's face and voice. It also uses an emotion recognition engine to analyze the user's emotions and generate appropriate responses and comments. The device displays these responses and comments, along with suggestions and solutions based on the emotion analysis, and receives the user's responses.

[0732] 3. Meeting Management Support:

[0733] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or if time is running out. Furthermore, if the user's emotions change in a negative direction, it generates suggestions and measures to ease the atmosphere. These reminders and suggestions are displayed to the user on the device, and the user's responses and actions are sent to the server.

[0734] 4. Meeting logging and feedback:

[0735] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[0736] Specific examples

[0737] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[0738] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, it is possible to identify areas for improvement for future meetings.

[0739] The processing flow will be explained below.

[0740] Step 1:

[0741] server:

[0742] Receive advance notice of upcoming meetings, obtain a list of participants, and analyze each participant's speaking habits and role by referencing a database of past meetings. This allows you to estimate in advance what each participant will contribute and be prepared to provide appropriate support.

[0743] Device:

[0744] When a meeting starts, the video chat software launches. The user enters their account information and sends the authentication information to the server. This allows the software to provide optimal assistance based on each user's settings and past speaking habits.

[0745] User:

[0746] Users input their speaking preferences and AI behavior settings into the interface, such as "don't interrupt me" and "provide timely advice."

[0747] Step 2:

[0748] server:

[0749] The system analyzes video and audio feeds in real time, recognizes the speaker's face and voice, and then uses an emotion recognition engine to analyze the user's emotions. Based on this, appropriate responses and comments are generated and sent to the device. For example, if User A is nervous while speaking, the generated comments might include something like "Relax."

[0750] Device:

[0751] The system displays the backchannels and comments received from the server to the user. It also displays suggestions and measures based on emotion analysis. The system sends the user's responses and actions to the server, enabling real-time feedback.

[0752] User:

[0753] The user can review the AI's suggestions and choose to accept or ignore them. For example, if the AI ​​suggests "Let's move on to the next agenda item," the user can choose to accept or ignore the suggestion.

[0754] Step 3:

[0755] server:

[0756] Monitor the progress of the meeting. Generate reminders if the agenda is not progressing smoothly or if there is little time left in the meeting. Also, if the user's emotions change to a negative direction, generate suggestions or measures to ease the atmosphere. For example, if it feels like the meeting is dragging on, generate a suggestion such as "Let's take a break."

[0757] Device:

[0758] It displays reminder messages and suggestions to users and sends user responses and actions to the server, ensuring smooth progress of the meeting.

[0759] User:

[0760] The user can check the reminder messages and suggestions and adjust the progress of the meeting as necessary, for example, by following the suggestions to "move on to the next agenda item" or "spend more time on the current agenda item."

[0761] Step 4:

[0762] server:

[0763] Record a meeting log, save what was said and the progress of the agenda, and generate an analysis report after the meeting, which can be used for future meetings.

[0764] Device:

[0765] After the meeting, the recorded logs and generated analysis reports are provided to the user. The user can also edit and save the logs as needed, making record management easier.

[0766] User:

[0767] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance, helping to identify areas for improvement and optimizing the system for future use.

[0768] Step 5:

[0769] server:

[0770] The system receives feedback from users and makes adjustments to be applied to future meetings, thereby enabling continuous improvement of the system.

[0771] Device:

[0772] For your next meeting, we'll optimize your settings based on accumulated data and user feedback, ensuring effective support for future meetings.

[0773] User:

[0774] Users can use the saved meeting logs and analysis data to prepare for their next meeting, thereby improving the efficiency and effectiveness of meetings.

[0775] Example 2

[0776] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0777] In online meetings, it is difficult to properly manage the emotions of speakers and the progress of the meeting in real time. Particularly when there are many participants, it is necessary to grasp the emotions and progress of each speaker and automatically provide appropriate responses and reminders, but conventional systems have not been able to achieve this. Therefore, improving the efficiency of meetings and ensuring smooth communication between participants is a challenge.

[0778] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0779] In this invention, the server includes means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the speaker's face and perform voice analysis, means for analyzing the user's emotions using an emotion recognition engine, generating appropriate backchannels and comments, and presenting them to the user in real time, means for monitoring the progress of the conference and generating reminders when progress on the agenda is not being made or time is running out, and means for recording a conference log and saving the content of comments and the progress of the agenda. This makes it possible to grasp user emotions and the progress of the conference in real time and respond appropriately.

[0780] A "participant list" is a list that includes the names and IDs of all users who will be participating in a conference.

[0781] A "conference database" is a database that stores information about past conferences, and stores data on what was said and the roles of participants.

[0782] "Speech trends" are behavioral patterns obtained by analyzing the content, frequency, and role of each participant in past meetings.

[0783] "Facial recognition" is a technology that analyzes facial images obtained from a video feed and identifies the speaker's face.

[0784] "Speech analysis" is a technology that analyzes audio feeds and converts what is being said into text.

[0785] An "emotion recognition engine" is an engine that analyzes video and audio data to detect a user's emotional state (e.g., joy, sadness, tension, etc.).

[0786] "Responses and comments" refer to feedback and reactions generated based on what is said, and are elements that help smooth the flow of conversation.

[0787] "Reminders" are notifications and alerts about meeting progress and time management, designed to prevent delays and time-outs.

[0788] A "meeting log" refers to a record of all data such as what was said during the meeting, the progress, and the degree to which the agenda was achieved.

[0789] "Suggestions and measures" are specific actions generated based on emotion analysis to improve the user's condition or the progress of the meeting.

[0790] This invention is a "Video Chat AI" system for facilitating communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions of the server and terminals to efficiently and effectively conduct meetings.

[0791] 1. User Awareness and Initial Setup:

[0792] server:

[0793] The server receives advance notice of the meeting and retrieves the attendee list, for example, using the Google Calendar API to retrieve the meeting schedule.

[0794] The server retrieves relevant data from a past conference database (e.g., MySQL) and analyzes participants' speaking tendencies and roles.

[0795] Device:

[0796] When a user joins a meeting, video chat software (e.g., Zoom, Microsoft Teams) is launched.

[0797] The user enters their account information and sends their authentication information to the server to log in.

[0798] Users input their preferred speaking style and AI behavior settings into the interface.

[0799] 2. Analyzing video and audio feeds:

[0800] server:

[0801] The server analyzes the video and audio feeds in real time, using OpenCV for face recognition of the speaker and the Google Cloud Speech-to-Text API for speech analysis.

[0802] An emotion recognition engine (e.g., IBM Watson Emotion Recognition) is used to analyze the user's emotions and generate appropriate responses and comments.

[0803] Device:

[0804] Display received responses and comments to the user.

[0805] Suggestions and measures based on sentiment analysis are displayed to the user.

[0806] 3. Meeting Management Support:

[0807] server:

[0808] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or time is running out.

[0809] Device:

[0810] Display reminder messages and suggestions to the user and send the user's responses and actions to the server.

[0811] 4. Meeting logging and feedback:

[0812] server:

[0813] The server records the meeting log and stores the content of the discussion and the progress of the agenda in a database.

[0814] After the meeting, the generated logs and analysis reports are provided to the user so that they can be used in future meetings.

[0815] Device:

[0816] Users can review meeting recordings and logs and edit them as needed.

[0817] Users can provide feedback and rate the AI's performance to identify areas for improvement in future meetings.

[0818] Specific examples

[0819] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[0820] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, improvements can be made to future meetings.

[0821] Prompt Sentence Examples

[0822] Here are some example prompts to input to the generative AI model:

[0823] I have a question about the functionality of the "Video Chat AI" system I will be using in my next online meeting. Could you please explain in detail how this system, which is equipped with an emotion recognition engine, analyzes the user's emotions and generates appropriate responses and comments? Also, could you please explain in detail how it will support the progress of the meeting?

[0824] This system ensures smooth progress of meetings and manages participants' emotions.

[0825] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0826] Step 1: Prepare for the meeting

[0827] server:

[0828] The server receives advance notices of upcoming meetings. As input, meeting schedule information is provided, e.g., data retrieved from the Google Calendar API.

[0829] The server generates a participant list based on this input data and retrieves data related to each participant from a past conference database (e.g., MySQL). The output is a participant list and the results of an analysis of past speaking tendencies and roles.

[0830] Device:

[0831] The device will send the generated participant list and meeting notification to the user via email or in-app notification.

[0832] Specific behavior:

[0833] The attendee list and meeting materials are displayed on the screen for easy user confirmation, and meeting notifications are sent via email and in-app notifications.

[0834] Step 2: Start the meeting and authenticate users

[0835] server:

[0836] The server receives a signal to start a conference, and inputs include a conference start instruction and each user's authentication information.

[0837] The server connects to a SaaS (Software as a Service) service (e.g., Zoom, Microsoft Teams) to start the conference. The output is the conference room URL and the user authentication result.

[0838] Device:

[0839] The terminal starts the video chat software and sends the user's account information to the server, with the input data being the user's authentication information.

[0840] If the authentication information is successful, the terminal displays the conference room URL returned by the server. The output is the result of providing the conference room URL and verifying the settings information.

[0841] User:

[0842] Users receive a URL or code for joining the meeting and join the meeting.

[0843] The user inputs their preferred speech style and AI behavior settings into the interface. The input data is user preference information, which is then sent to the server.

[0844] Specific behavior:

[0845] The server processes the authentication information and, if successful, provides the device with a meeting room URL, which the user can use to join the meeting in real time.

[0846] Step 3: Analyze the video and audio feeds

[0847] server:

[0848] The server analyzes the video and audio feeds in real time. The input data are the video and audio feeds sent from the devices.

[0849] The speaker's face is recognized using OpenCV, and speech analysis is performed using the Google Cloud Speech-to-Text API. Emotions are analyzed using an emotion recognition engine (e.g., IBM Watson Emotion Recognition). The output is the speaker's face position, speech analysis results, and emotion analysis results.

[0850] Device:

[0851] Displays the backchannels and comments received from the server. The input data is the analysis results sent from the server. The output is the display of feedback to the user.

[0852] Specific behavior:

[0853] When the user starts speaking, the device sends video and audio data to the server, which returns the analysis results to the device and displays them to the user.

[0854] Step 4: Sentiment analysis and response

[0855] server:

[0856] An emotion recognition engine is used to analyze the user's emotions, and the input data is video and audio feed from the device.

[0857] Based on the results of the sentiment analysis, appropriate responses and comments are generated and sent to the device. The output is the sentiment analysis results and the comments and responses based on them.

[0858] Device:

[0859] The received comments and responses are displayed to the user. The input data is the emotion analysis results sent from the server. The output is the content displayed to the user.

[0860] Specific behavior:

[0861] If the user is nervous, display a pop-up notification such as "Would you like to take a short break?"

[0862] Step 5: Managing the meeting

[0863] server:

[0864] The server monitors the progress of the conference, and input data includes the progress status and time information of the conference.

[0865] If the agenda is not progressing or time is running out, a reminder is generated and sent to the terminal. The output is the reminder message.

[0866] Device:

[0867] Display reminders received from the server to the user. The input data is the reminder message sent from the server. The output is the notification content to the user.

[0868] Specific behavior:

[0869] When there is little time left, a notification such as "10 minutes remaining" will be displayed.

[0870] Step 6: Meeting logging and feedback

[0871] server:

[0872] The server records the conference log, and input data includes the content of the conference and its progress.

[0873] The system stores the content of comments and the progress of the agenda in a database, and provides the user with logs and analysis reports for use in future meetings. The output is a meeting log and analysis report.

[0874] Device:

[0875] The conference log and analysis report received from the server are displayed so that the user can check them. The input data are the conference log and analysis report. The output is the content displayed to the user.

[0876] It provides an interface for users to input feedback and evaluate the AI's performance. The input is the user's feedback. The output is the feedback result.

[0877] Specific behavior:

[0878] After the meeting, the recording and log of the meeting are provided to the user, who can review them and provide feedback. This will help evaluate the AI's performance and help improve it for future meetings.

[0879] (Application example 2)

[0880] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0881] Analyzing employee conversations and emotions in real time and providing appropriate feedback during factory meetings and conferences is a difficult task. It is particularly important to ensure that meetings proceed smoothly and create an environment where employees can participate without feeling nervous or stressed. It is also necessary to efficiently manage meeting logs and recordings and analyze the information for use in future meetings.

[0882] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0883] In this invention, the server includes: means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referencing a past conference database; means for analyzing video and audio feeds to recognize the speaker's face and perform voice analysis; means for generating appropriate backchannels and comments based on the content of the speech and presenting them to the user in real time; and means for analyzing user emotions in real time using an emotion recognition engine and generating proposals and countermeasures based on the emotion analysis results. This enables real-time and appropriate feedback and emotion management in factory conferences and meetings. It also enables smooth progress in meetings and provides an environment where employees can participate with peace of mind. Furthermore, the recorded meeting logs and recordings can be used to improve future meetings.

[0884] A "participant list" is a list of people participating in a conference or meeting, and includes information for understanding the role and past speaking tendencies of each participant.

[0885] A "conference database" is a collection of information that stores data such as logs, audio recordings, and video recordings of past meetings, and is used to analyze the speaking tendencies and roles of each participant.

[0886] "Speech tendency" refers to a pattern that indicates what each participant talks about and how often they speak in a conference or meeting.

[0887] "Roles" indicate the position and scope of responsibility of each participant in a conference or meeting, and include, for example, leader, administrative staff, technical staff, etc.

[0888] A "video feed" is a stream of video data that broadcasts video of conference or meeting participants in real time.

[0889] An "audio feed" is a stream of audio data that delivers the voices of conference or meeting participants in real time.

[0890] "Facial recognition" is a technology that analyzes video data to identify specific people, and is used to identify speakers.

[0891] "Voice analysis" is a technology that analyzes voice data to understand what is being said, and is used to generate appropriate responses and comments.

[0892] An "emotion recognition engine" is software or hardware for analyzing a user's emotional state from facial expressions, voice, etc.

[0893] "Ai-chi" is a short phrase or expression used during a conversation to show understanding and empathy for what the other person is saying, and is used to facilitate communication.

[0894] A "comment" is a word used to express an appropriate opinion or reaction to something said during a conference or meeting.

[0895] A "proposal" is a specific idea or action point that encourages new opinions or actions based on what was said during a conference or meeting.

[0896] "Countermeasures" are specific actions or methods proposed to solve the problem based on the content of the comments and the results of emotion analysis.

[0897] A "log" is data that records what is said during a conference or meeting, the progress of the agenda, etc.

[0898] "Recording" refers to data that records video and audio of a conference or meeting.

[0899] An "analysis report" is a document that evaluates the effectiveness and progress of a meeting based on recorded logs and video data, and summarizes the analysis results to be used in future meetings.

[0900] "Reminds" are notifications and alerts that monitor the progress of meetings and conferences and prompt necessary action if things are not going as planned or time is running out.

[0901] This invention provides a "communication support robot system" for smoothly conducting conferences and meetings within a factory. How the system is implemented will be explained below in detail.

[0902] System Overview

[0903] This system mainly consists of a server, robot terminals, and users. The server is responsible for storing and analyzing meeting data and for real-time communication. The robot terminals are devices that allow direct interaction with users within the factory and are equipped with sensors such as cameras and microphones.

[0904] Server Features

[0905] 1. Participant list acquisition and conference database reference function:

[0906] The server receives advance notice of the meeting and a list of participants, and analyzes each participant's speaking tendencies and roles by referring to a database of past meetings.

[0907] 2. Video and audio feed analysis:

[0908] The server analyzes the video and audio feeds sent from the robot terminal in real time, performs facial recognition and voice analysis of the speaker, and analyzes the user's emotions using an emotion recognition engine (e.g., Microsoft's Azure Cognitive Services).

[0909] 3. Sentiment analysis-based suggestion and countermeasure generation function:

[0910] The server generates appropriate responses and comments based on the analysis results, and if necessary, suggests ways to relax. This data is then sent to the robot terminal in real time.

[0911] 4. Meeting progress support function:

[0912] The server monitors the progress of the meeting and generates reminders if the meeting is not progressing as planned or if time is running out.

[0913] 5. Meeting logging and analysis report generation function:

[0914] The server records the meeting log, saving what was said and the progress of the agenda, and generates an analytical report for later review.

[0915] Robot terminal functions

[0916] 1. User input and authentication:

[0917] When the meeting starts, the robot terminal launches the video chat software, the user enters account information into the interface, and the authentication information is sent to the server.

[0918] 2. Get real-time feeds:

[0919] The robot terminal uses a camera and microphone to capture the user's video and audio feeds in real time and transmits them to a server.

[0920] 3. Display of suggestions and measures:

[0921] The system displays to the user the responses, comments, and suggestions and measures based on sentiment analysis sent from the server, and receives the user's responses.

[0922] 4. Processing example:

[0923] For example, if a leader is detected to be nervous during a presentation at a quality control meeting in a factory, the robot will respond by saying, "It's okay, stay calm. What I've just explained is very important."

[0924] The robot will also detect if the agenda is not progressing during a meeting and display a reminder, saying, "We need to confirm the next step on this topic."

[0925] Prompt Sentence Examples

[0926] "If you sense tension in a meeting, a prompt will suggest:

[0927] Generate suggestions to ease the leader's tension by saying, "It's okay to stay calm. What I just explained is very important."

[0928] In this way, the communication support robot system can smoothly conduct conferences and meetings within factories and provide real-time support for employee emotional management, thereby improving meeting efficiency and reducing stress for participants.

[0929] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0930] Step 1:

[0931] Obtaining and pre-analyzing the participant list

[0932] Before the meeting starts, the server receives advance notice of the meeting, obtains the participant list, and then analyzes each participant's speaking habits and roles by referring to a database of past meetings.

[0933] Input: Meeting advance notices, participant list, past meeting database

[0934] Data processing and calculation: The server reads the conference database and compares it with the participant list, analyzing the frequency of each participant's comments and topic trends.

[0935] Output: Analysis results (commentary trends, roles)

[0936] Step 2:

[0937] Conference start and user authentication

[0938] When the conference starts, the terminal launches the video chat software, prompts the user to enter account information, and transmits the authentication information to the server.

[0939] Input: User account information, authentication information

[0940] Data processing and data calculation: The terminal sends the entered account information to the server, and the server verifies the authentication information and confirms the user.

[0941] Output: Authentication result (authentication success / failure)

[0942] Step 3:

[0943] Acquiring and sending video and audio feeds

[0944] The device uses a camera and microphone to capture the user's video and audio feeds in real time during the meeting and transmits them to the server.

[0945] Input: Video feed, Audio feed

[0946] Data processing and data calculation: The device sends the acquired video and audio feeds to the server.

[0947] Output: Video feed and audio feed sent

[0948] Step 4:

[0949] Real-time video and audio analysis

[0950] The server analyzes the received video and audio feeds in real time, performs face recognition and voice analysis of the speaker, and also analyzes the user's emotions using an emotion recognition engine.

[0951] Input: Video feed, Audio feed

[0952] Data processing and data calculation: The server uses a facial recognition algorithm and a voice analysis engine to identify the speaker, and an emotion recognition engine to analyze the user's emotions.

[0953] Output: Face recognition results, voice analysis results, emotion analysis results

[0954] Step 5:

[0955] Generate and present backchannels and comments

[0956] The server generates appropriate responses and comments based on the content of the speech and the results of emotion analysis, and sends them to the device, which then displays them to the user and outputs them as audio.

[0957] Input: Face recognition results, voice analysis results, emotion analysis results

[0958] Data processing and data calculation: The server uses the generative AI model to generate appropriate responses and comments, and sends them to the device.

[0959] Output: backchannel, comments

[0960] Step 6:

[0961] Meeting progress monitoring and reminder generation

[0962] The server monitors the progress of the meeting in real time, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which then notifies the user.

[0963] Input: Meeting progress (log, timestamp)

[0964] Data processing and data calculation: The server analyzes the progress, generates reminders and sends them to the device.

[0965] Output: Reminder notification

[0966] Step 7:

[0967] Recommendation generation based on emotion changes

[0968] If the user's emotional state changes to a negative state, the server generates relaxation suggestions and sends them to the device, which displays them to the user and also outputs them audibly.

[0969] Input: Sentiment analysis results

[0970] Data processing and data calculation: The server uses the generative AI model to generate relaxation suggestions and send them to the device.

[0971] Output: Proposal notification

[0972] Step 8:

[0973] Meeting logging and analytical report generation

[0974] The server records the meeting log, saving the content of comments and the progress of the agenda. After the meeting, an analysis report is generated based on this data, allowing users to use the results in future meetings.

[0975] Input: Meeting log (contents, progress, timestamp)

[0976] Data processing and data calculation: The server stores the log data and generates an analysis report after completion.

[0977] Output: Log data, analysis report

[0978] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0979] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0980] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0981] [Third embodiment]

[0982] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0983] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0985] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0989] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0993] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0994] This invention provides a system called "Video Chat AI" that facilitates smooth communication during online meetings. This system links multiple functions on the server and terminal sides to efficiently and effectively conduct meetings.

[0995] System Overview

[0996] server:

[0997] The server receives advance notification of the meeting and a list of participants, and refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes the video and audio feeds during the meeting in real time, performing facial recognition and audio analysis of the speakers. It generates backchannels and comments in real time and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[0998] Device:

[0999] The device displays the user's responses and comments received from the server in real time, sends the user's responses and operations to the server, and records the meeting log, providing a saving and editing function so that the user can check it after the meeting has ended.

[1000] A natural language explanation of the program's processing

[1001] 1. User Awareness and Initial Setup:

[1002] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device side, the video chat software launches when the meeting starts, and the user enters their account information and sends authentication information to the server.

[1003] 2. Analyzing video and audio feeds:

[1004] The server analyzes the video and audio feeds in real time, performs facial recognition and speech analysis of the speaker, generates appropriate responses and comments, and sends them to the device, where they are displayed to the user and receive their responses.

[1005] 3. Meeting Management Support:

[1006] The server monitors the progress of the meeting, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which displays it to the user so that the user can smoothly proceed with the meeting.

[1007] 4. Meeting logging and feedback:

[1008] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[1009] Specific examples

[1010] For example, consider the case where User A participates in an online conference. The server analyzes User A's past speaking habits in advance, and generates appropriate backchannels and comments while analyzing User A's comments in real time during the conference. As User A continues to speak, backchannels sent from the server (e.g., "I see, that's interesting") are displayed on the device, smoothly advancing the conversation. Furthermore, as the conference draws to a close, the server generates a reminder saying, "There are 10 minutes left," and the device notifies User A of this, allowing the conference to proceed efficiently.

[1011] This allows users to maintain natural communication during online meetings and ensure important points are covered. Furthermore, after the meeting, users can identify areas for improvement for the next meeting through log recording and feedback, enabling continuous improvement.

[1012] The processing flow will be explained below.

[1013] Step 1:

[1014] server:

[1015] Receive advance notice of meetings, obtain a list of participants, and analyze each participant's speaking habits and roles by referencing a database of past meetings.

[1016] Device:

[1017] When the meeting begins, the video chat software launches, the user enters their account information, and the authentication information is sent to the server.

[1018] User:

[1019] Users input their speaking preferences and AI behavior settings into the interface.

[1020] Step 2:

[1021] server:

[1022] It analyzes video and audio feeds in real time, performs facial recognition and audio analysis of the speaker, and generates appropriate responses and comments.

[1023] Device:

[1024] Displays the user any backchannels or comments received from the server. Sends the user's responses and actions to the server.

[1025] User:

[1026] The user can review the AI's suggestions and choose to accept or ignore them. The AI ​​responds with feedback and additional information in real time while the user is speaking.

[1027] Step 3:

[1028] server:

[1029] Monitor meeting progress and generate reminders if the agenda is falling behind or if the meeting is running out of time.

[1030] Device:

[1031] Display a reminder message to the user. Send the user's response or operation to the server.

[1032] User:

[1033] The user checks the reminder message and adjusts the progress of the meeting as necessary.

[1034] Step 4:

[1035] server:

[1036] Record a meeting log, saving what was said and what was discussed, and generating an analytical report after the meeting.

[1037] Device:

[1038] After the meeting, the recorded log and generated analysis report are provided to the user, and the user can edit and save the log as needed.

[1039] User:

[1040] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance.

[1041] Step 5:

[1042] server:

[1043] Receive feedback from users and make adjustments to be utilized in future meetings.

[1044] Device:

[1045] Optimize settings for your next meeting based on accumulated data and user feedback.

[1046] User:

[1047] Users can use the saved meeting logs and analysis data to prepare for their next meeting.

[1048] Example 1

[1049] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] In online meetings, communication between participants can sometimes be disrupted. Participants often lose their ability to speak, or the agenda often doesn't move forward. Furthermore, recording and post-meeting analysis of meeting content is often inadequate, resulting in a lack of feedback to improve future meetings. A solution to these problems is needed.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1052] In this invention, the server includes a means for acquiring identification information for the conference and analyzing the behavioral tendencies and roles of each participant by referring to past information records, a means for analyzing image and audio feeds to perform face recognition and acoustic analysis of speakers, a means for generating appropriate reactions and comments based on the content of comments and presenting them to participants in real time, a means for monitoring the progress of the conference and issuing warnings if the agenda is not progressing or time is running out, a means for saving the conference record and generating analysis results to utilize the content of comments and the progress of the agenda for future conferences, and a function for transmitting the generated reactions and comments to terminals and presenting them to conference participants in real time. This enables smooth communication in online conferences and efficient recording and analysis of conference content.

[1053] "Conference identification information" is data that identifies a particular conference and includes information about participants and agenda items.

[1054] "Past information records" are data that record the content of statements and progress of previous meetings.

[1055] "Means for analyzing behavioral tendencies and roles" is a function that analyzes the characteristics and roles of each participant based on their past comments and participation status.

[1056] An "image feed" is video data transmitted in real time from a video device such as a camera.

[1057] An "acoustic feed" is audio data transmitted in real time from an audio device such as a microphone.

[1058] "Speaker facial recognition" is a technology that analyzes video data to identify specific people.

[1059] The means of "acoustic analysis" is a function that analyzes audio data to understand the speaker's voice and content.

[1060] "Means for generating reactions and mentions" is a function that automatically creates responses and comments based on the analyzed content of statements.

[1061] "Means for monitoring the progress of a meeting" refers to a function that monitors the progress of agenda items and the allocation of speaking time in real time.

[1062] The "means of issuing a warning" is a function that notifies participants when the meeting is not progressing or when time is running out.

[1063] "Means for saving meeting records" refers to a function that records the contents of a meeting in real time and saves them for later reference.

[1064] The "means for generating analysis results" is a function that analyzes recorded meeting data and provides useful feedback for future meetings.

[1065] "Having the ability to present in real time" refers to technology that allows participants to instantly see their reactions and comments.

[1066] The present invention provides a system for facilitating communication during online conferences. In this system, a server and terminals work in cooperation with each other to support the progress of the conference. Specific embodiments of the system are described below.

[1067] Server Operation

[1068] The server receives advance notice of the meeting, retrieves the list of participants, and analyzes the behavioral trends and roles of each participant by referencing past information records. For this purpose, a database search engine or machine learning model can be used.

[1069] After the meeting starts, the server receives and analyzes the image and audio feeds in real time. It uses facial recognition and acoustic analysis technologies to identify the speaker and analyze the content of their speech. Specifically, OpenCV is used for facial recognition and IBM Watson Speech to Text API is used for audio analysis.

[1070] The server generates appropriate responses and comments based on the content of the comment and sends them to the device in real time using a generative AI model (e.g., GPT-3).

[1071] In addition, the server monitors the progress of the meeting and generates alerts when the agenda is not progressing or time is running out, which are also sent to the terminal in real time and notified to the user.

[1072] After the meeting ends, the server saves the meeting record and generates analysis results based on the content of the comments and the progress of the agenda. These analysis results are provided to the user for use in future meetings.

[1073] Device behavior

[1074] The terminal receives input from the user and sends it to the server. When a conference starts, the terminal launches the video chat software, prompts the user to enter their account information, and sends the authentication information to the server. After authentication, the conference interface is displayed.

[1075] During the meeting, real-time reactions and comments sent from the server are displayed, allowing users to maintain natural conversations. Alerts from the server are also displayed to support the progress of the meeting.

[1076] After the meeting, the device provides the user with a recorded meeting log and recording, which the user can review and edit as necessary.

[1077] Specific examples

[1078] When User A participates in an online conference, the server analyzes User A's past speaking habits in advance and generates appropriate responses and comments. For example, when User A makes a comment during a conference, the server generates a response such as "That's interesting. Please tell us more about it," and sends it to the device. The device displays this to User A in real time, smoothly advancing the conversation.

[1079] As the meeting draws to a close, the server generates a warning such as "10 minutes remaining," and the terminal notifies User A of this, thereby supporting time management.

[1080] This allows User A to maintain natural communication during online meetings and efficiently progress important agenda items. In addition, the logs and feedback recorded after the meeting allow User A to identify areas for improvement in the next meeting.

[1081] Example prompts for generative AI models

[1082] Here are some example prompts you can enter into your generative AI model:

[1083] In an online conferencing system using "video chat AI," the server identifies the speaker in real time during the meeting and generates appropriate responses and comments. Please explain the specific operations at each step of this system, including the user authentication process and the specific processing of real-time analysis during the meeting.

[1084] Using this prompt, the generative AI model can provide detailed explanations and concrete examples.

[1085] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1086] The flow of this system's program processing

[1087] Step 1: User authentication

[1088] Step 2: Prepare for the meeting

[1089] Step 3: Real-time analytics during the meeting

[1090] Step 4: Meeting facilitating support

[1091] Step 5: Record the meeting and provide feedback

[1092] Specific explanation of each processing step

[1093] Step 1: User authentication

[1094] Input: User account information (email address, password)

[1095] Specific behavior:

[1096] The server waits for the user to launch the video chat software and enter their account information. When the user enters their email address and password, the device sends this to the server, which then compares the received information with a database of past authentications and authenticates the user.

[1097] Data processing and calculation:

[1098] The server hashes the entered account information and compares it with an existing database. If authentication is successful, the server sends an authentication success message to the terminal.

[1099] Output: Authentication result (success / failure), if successful, the conference interface is displayed

[1100] Step 2: Prepare for the meeting

[1101] Input: Participant list, past conference database

[1102] Specific behavior:

[1103] The server receives advance notice of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze the behavioral trends and roles of each participant. The user checks the start time of the meeting and makes any necessary preparations. The server then sends a reminder before the meeting.

[1104] Data processing and calculation:

[1105] Based on the participant list, past meeting data is extracted from the database, and an analytical algorithm is used to analyze behavioral trends and roles. The analysis results are stored on the server and used for real-time analysis during the meeting.

[1106] Output: Participant behavioral trends and roles, reminder notifications

[1107] Step 3: Real-time analytics during the meeting

[1108] Input: Video feed, audio feed

[1109] Specific behavior:

[1110] When a meeting begins, the server acquires and analyzes video and audio feeds in real time. It performs facial recognition and audio analysis of the speaker, generates appropriate responses and comments, and sends them to the device. The user can view these on the screen. Specifically, it uses OpenCV for facial recognition and IBM Watson Speech to Text API for audio analysis.

[1111] Data processing and calculation:

[1112] Facial recognition algorithms are used to identify speakers from video feeds, and speech recognition technology is used to convert speech into text from audio feeds, which is then used to generate real-time responses and comments using generative AI models (e.g., GPT-3).

[1113] Output: Real-time reactions and comments (sent to terminal)

[1114] Step 4: Meeting facilitating support

[1115] Input: Meeting progress, time management data

[1116] Specific behavior:

[1117] The server monitors the progress of the meeting and generates a warning if the agenda is not progressing or if time is running out. This information is sent to the terminal in real time and notifies the user.

[1118] Data processing and calculation:

[1119] It analyzes progress and time management data in real time and generates alerts based on set conditions, such as when no progress is made on an agenda item for a certain period of time or when a meeting is 10 minutes away from the scheduled end time.

[1120] Output: Warning message (sent to terminal)

[1121] Step 5: Record the meeting and provide feedback

[1122] Input: Meeting video feed, audio feed, speech log

[1123] Specific behavior:

[1124] The server stores the meeting records and generates analysis results based on the content of comments and the progress of the agenda. After the meeting, users can check the logs and recordings and make any necessary edits.

[1125] Data processing and calculation:

[1126] It stores the video and audio feeds of the meeting and analyzes speech logs to generate insights that can be used for future meetings.

[1127] Output: Meeting logs, recordings, and analysis results (provided to users)

[1128] (Application example 1)

[1129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1130] With conventional online conference systems and virtual store systems, it was difficult to maintain smooth communication with participants and customers. In particular, important topics could not be advanced smoothly during the meeting or customer service process, leaving participants and customers frustrated. This led to a decline in meeting efficiency and customer service quality, ultimately resulting in a deterioration in overall business performance.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1132] In this invention, the server includes means for acquiring a participant list and analyzing the speaking tendencies and roles of each participant by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the face and perform audio analysis of the speaker, means for generating appropriate interjections and comments according to the content of the speech and presenting them to the user in real time, and means for analyzing the facial expressions and speech content of customers in real time and providing comments and advice on appropriate ways to respond, thereby enabling the smooth progress of the conference and customer service and smooth communication that satisfies users and customers.

[1133] A "participant list" is a list containing information about people who will be participating in a conference.

[1134] The "past conference database" is a collection of information that stores records and details of conferences that have been held in the past.

[1135] "Speech trends" indicate the characteristics of the content and patterns of speech that a particular participant often makes during a meeting.

[1136] "Roles" refer to the duties and responsibilities of each participant in a meeting.

[1137] A "video feed" is a continuous stream of video data captured through a camera.

[1138] An "audio feed" is a continuous stream of audio data captured through a microphone.

[1139] "Facial recognition" is a technology that analyzes images captured by a camera to identify the face of a specific individual.

[1140] "Voice analysis" is a technology that analyzes the content of speech and the characteristics of the speaker based on acquired voice data.

[1141] "Ai-chi" refers to a reaction or response that shows that you are listening.

[1142] A "comment" is an opinion or additional information about a particular statement.

[1143] "Customer" means a consumer who visits the virtual store to purchase products or services.

[1144] "Facial expressions" are the movements and changes that appear on a person's face to show their emotions.

[1145] "Utterance content" refers to the specific content that is spoken in a conversation or statement.

[1146] "Response methods" refer to the specific means or methods of responding to customers or participants.

[1147] "Comments and Advice" means advice or opinions given based on statements or circumstances.

[1148] "Customer service" refers to the act of explaining and selling products and services to customers.

[1149] A "remind" is a notification or instruction given to remind you of a specific thing.

[1150] "Meeting progress" indicates the progress of the meeting agenda or topic.

[1151] "Logging" means recording the details of what is said and what happens during a meeting or customer service session.

[1152] An "analysis report" is a report summarizing the results of an analysis based on recorded data.

[1153] This invention provides a system that enables smooth communication in virtual stores and online meetings. Specifically, the server and terminals work together to analyze the facial expressions and speech of customers and participants in real time, and provide appropriate responses and advice.

[1154] Hardware and software used

[1155] 1. Hardware:

[1156] Smart glasses (e.g., Google Glass, Microsoft HoloLens)

[1157] Server (e.g. AWS, Google Cloud Platform)

[1158] 2. Software:

[1159] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1160] Facial recognition API (e.g. Microsoft Azure Face API)

[1161] Real-time communication libraries (e.g., WebRTC)

[1162] System Configuration

[1163] The server has the following means:

[1164] 1. A means of obtaining a list of participants and analyzing each participant's speaking tendencies and role by referencing a database of past conferences.

[1165] 2. A means of analyzing the video and audio feeds to perform facial recognition and voice analysis of the speaker.

[1166] 3. A means of generating appropriate responses and comments based on what is being said and presenting them to the user in real time.

[1167] 4. A means of analyzing customers' facial expressions and speech in real time and providing comments and advice on how to respond appropriately.

[1168] The terminal has the following means:

[1169] 1. A means of displaying backchannels and comments received from the server and sending user responses and actions to the server.

[1170] 2. A means of recording past customer service data and providing the ability to review it later.

[1171] Program processing explanation

[1172] The server receives the login information sent by the user from the smart glasses, analyzes the customer's speech patterns and facial expression patterns by referring to past customer service data, analyzes real-time video and audio feeds, and uses facial expression and voice recognition APIs to understand the customer's emotions and speech content. It then generates appropriate responses and advice and sends them to the smart glasses.

[1173] The smart glasses display the responses and advice sent from the server on the device. As the user continues the conversation with the customer, they respond according to the information displayed on the smart glasses. After the conversation is over, the acquired data is sent back to the server and a feedback report is generated for the next time.

[1174] Specific examples

[1175] For example, if staff member A wears smart glasses while serving customers in a virtual store, the server will analyze customer B's purchase history and preferences based on past data. When customer B visits the store, the smart glasses will display advice such as, "Please recommend new products related to products that customer B previously purchased." If customer B asks a question or looks gloomy, the server will analyze that information and provide staff member A with a comment such as, "The customer seems anxious. Please provide a reassuring response."

[1176] Prompt Sentence Examples

[1177] Analysis of customer conversations in a virtual store:

[1178] Perform facial recognition and sentiment analysis on Customer B to generate an appropriate response in real time.

[1179] If Customer B asks, "Can you tell me more about this product?" you respond with, "Of course. The main features of this product are..."

[1180] Store associates use smart glasses to view advice and reminders and help communicate with customers.

[1181] This allows virtual store staff to provide appropriate customer service in real time, improving customer satisfaction.

[1182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1183] Step 1:

[1184] The server retrieves the participant list and analyzes each participant's speaking tendencies and roles by referencing a database of past conferences. The server receives the participant list and past conference data as input, performs statistical analysis based on which patterns for each participant are extracted, and generates speaking tendencies data and role information for each participant as output.

[1185] Step 2:

[1186] The server collects video and audio feeds and analyzes them in real time. As input, the video and audio feeds sent from the smart glasses are provided. These feeds are analyzed using facial expression recognition API and speech recognition API to perform facial recognition and speech analysis. As output, analyzed facial expression data and speech content data are obtained.

[1187] Step 3:

[1188] The server generates appropriate backchannels and comments based on the content of the speech. The speech content data obtained in step 2 is used as input. Based on this, appropriate backchannels and comments are generated using natural language processing (NLP) technology. The output is generated as an appropriate response or comment to the user.

[1189] Step 4:

[1190] The server analyzes the customer's facial expressions and speech content in real time, and generates comments and advice indicating the appropriate response method. The facial expression data and speech content data acquired in step 2 are used as input. Based on the analysis results, the server understands the customer's emotional state and requests, and uses a generative AI model to present the appropriate response method. Specific comments and advice are generated as output.

[1191] Step 5:

[1192] The device displays the backchannels, comments, and advice received from the server on the smart glasses. The backchannels, comments, and advice data sent from the server are used as input. This is displayed in real time on the smart glasses' display and provided to the user. The output is the information that the user actually views.

[1193] Step 6:

[1194] The terminal records past customer service data and generates a feedback report for the next time. Input includes user responses and conversations with customers. This data is recorded as a log and sent to the server. The server analyzes the data and compiles it into a feedback report to be used in the next customer service. As output, a feedback report is generated and provided to the user.

[1195] Through these steps, a system will be established that provides appropriate support for customer service in virtual stores in real time, improving customer satisfaction.

[1196] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1197] This invention provides a "Video Chat AI" system for smooth communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions on the server side and the terminal side to efficiently and effectively conduct meetings.

[1198] System Overview

[1199] server:

[1200] The server receives advance notification of the meeting and a list of participants, and then refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes video and audio feeds in real time, performing facial and audio analysis of speakers. It also analyzes the user's emotions using an emotion recognition engine, generates appropriate responses and comments, and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[1201] Device:

[1202] The device displays the user's responses and comments received from the server in real time, and sends the user's responses and actions to the server. It also displays suggestions and measures based on the results of user emotion analysis, and manages the atmosphere of the meeting. It also records the meeting log and provides saving and editing functions so that the user can review it after the meeting has ended.

[1203] A natural language explanation of the program's processing

[1204] 1. User Awareness and Initial Setup:

[1205] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device, the video chat software launches when the meeting begins, and the user enters their account information and sends authentication information to the server. The user then inputs their speaking preferences and AI behavior settings into the interface.

[1206] 2. Analyzing video and audio feeds:

[1207] The server analyzes the video and audio feeds in real time, recognizing the speaker's face and voice. It also uses an emotion recognition engine to analyze the user's emotions and generate appropriate responses and comments. The device displays these responses and comments, along with suggestions and solutions based on the emotion analysis, and receives the user's responses.

[1208] 3. Meeting Management Support:

[1209] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or if time is running out. Furthermore, if the user's emotions change in a negative direction, it generates suggestions and measures to ease the atmosphere. These reminders and suggestions are displayed to the user on the device, and the user's responses and actions are sent to the server.

[1210] 4. Meeting logging and feedback:

[1211] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[1212] Specific examples

[1213] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[1214] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, it is possible to identify areas for improvement for future meetings.

[1215] The processing flow will be explained below.

[1216] Step 1:

[1217] server:

[1218] Receive advance notice of upcoming meetings, obtain a list of participants, and analyze each participant's speaking habits and role by referencing a database of past meetings. This allows you to estimate in advance what each participant will contribute and be prepared to provide appropriate support.

[1219] Device:

[1220] When a meeting starts, the video chat software launches. The user enters their account information and sends the authentication information to the server. This allows the software to provide optimal assistance based on each user's settings and past speaking habits.

[1221] User:

[1222] Users input their speaking preferences and AI behavior settings into the interface, such as "don't interrupt me" and "provide timely advice."

[1223] Step 2:

[1224] server:

[1225] The system analyzes video and audio feeds in real time, recognizes the speaker's face and voice, and then uses an emotion recognition engine to analyze the user's emotions. Based on this, appropriate responses and comments are generated and sent to the device. For example, if User A is nervous while speaking, the generated comments might include something like "Relax."

[1226] Device:

[1227] The system displays the backchannels and comments received from the server to the user. It also displays suggestions and measures based on emotion analysis. The system sends the user's responses and actions to the server, enabling real-time feedback.

[1228] User:

[1229] The user can review the AI's suggestions and choose to accept or ignore them. For example, if the AI ​​suggests "Let's move on to the next agenda item," the user can choose to accept or ignore the suggestion.

[1230] Step 3:

[1231] server:

[1232] Monitor the progress of the meeting. Generate reminders if the agenda is not progressing smoothly or if there is little time left in the meeting. Also, if the user's emotions change to a negative direction, generate suggestions or measures to ease the atmosphere. For example, if it feels like the meeting is dragging on, generate a suggestion such as "Let's take a break."

[1233] Device:

[1234] It displays reminder messages and suggestions to users and sends user responses and actions to the server, ensuring smooth progress of the meeting.

[1235] User:

[1236] The user can check the reminder messages and suggestions and adjust the progress of the meeting as necessary, for example, by following the suggestions to "move on to the next agenda item" or "spend more time on the current agenda item."

[1237] Step 4:

[1238] server:

[1239] Record a meeting log, save what was said and the progress of the agenda, and generate an analysis report after the meeting, which can be used for future meetings.

[1240] Device:

[1241] After the meeting, the recorded logs and generated analysis reports are provided to the user. The user can also edit and save the logs as needed, making record management easier.

[1242] User:

[1243] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance, helping to identify areas for improvement and optimizing the system for future use.

[1244] Step 5:

[1245] server:

[1246] The system receives feedback from users and makes adjustments to be applied to future meetings, thereby enabling continuous improvement of the system.

[1247] Device:

[1248] For your next meeting, we'll optimize your settings based on accumulated data and user feedback, ensuring effective support for future meetings.

[1249] User:

[1250] Users can use the saved meeting logs and analysis data to prepare for their next meeting, thereby improving the efficiency and effectiveness of meetings.

[1251] Example 2

[1252] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1253] In online meetings, it is difficult to properly manage the emotions of speakers and the progress of the meeting in real time. Particularly when there are many participants, it is necessary to grasp the emotions and progress of each speaker and automatically provide appropriate responses and reminders, but conventional systems have not been able to achieve this. Therefore, improving the efficiency of meetings and ensuring smooth communication between participants is a challenge.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1255] In this invention, the server includes means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the speaker's face and perform voice analysis, means for analyzing the user's emotions using an emotion recognition engine, generating appropriate backchannels and comments, and presenting them to the user in real time, means for monitoring the progress of the conference and generating reminders when progress on the agenda is not being made or time is running out, and means for recording a conference log and saving the content of comments and the progress of the agenda. This makes it possible to grasp user emotions and the progress of the conference in real time and respond appropriately.

[1256] A "participant list" is a list that includes the names and IDs of all users who will be participating in a conference.

[1257] A "conference database" is a database that stores information about past conferences, and stores data on what was said and the roles of participants.

[1258] "Speech trends" are behavioral patterns obtained by analyzing the content, frequency, and role of each participant in past meetings.

[1259] "Facial recognition" is a technology that analyzes facial images obtained from a video feed and identifies the speaker's face.

[1260] "Speech analysis" is a technology that analyzes audio feeds and converts what is being said into text.

[1261] An "emotion recognition engine" is an engine that analyzes video and audio data to detect a user's emotional state (e.g., joy, sadness, tension, etc.).

[1262] "Responses and comments" refer to feedback and reactions generated based on what is said, and are elements that help smooth the flow of conversation.

[1263] "Reminders" are notifications and alerts about meeting progress and time management, designed to prevent delays and time-outs.

[1264] A "meeting log" refers to a record of all data such as what was said during the meeting, the progress, and the degree to which the agenda was achieved.

[1265] "Suggestions and measures" are specific actions generated based on emotion analysis to improve the user's condition or the progress of the meeting.

[1266] This invention is a "Video Chat AI" system for facilitating communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions of the server and terminals to efficiently and effectively conduct meetings.

[1267] 1. User Awareness and Initial Setup:

[1268] server:

[1269] The server receives advance notice of the meeting and retrieves the attendee list, for example, using the Google Calendar API to retrieve the meeting schedule.

[1270] The server retrieves relevant data from a past conference database (e.g., MySQL) and analyzes participants' speaking tendencies and roles.

[1271] Device:

[1272] When a user joins a meeting, video chat software (e.g., Zoom, Microsoft Teams) is launched.

[1273] The user enters their account information and sends their authentication information to the server to log in.

[1274] Users input their preferred speaking style and AI behavior settings into the interface.

[1275] 2. Analyzing video and audio feeds:

[1276] server:

[1277] The server analyzes the video and audio feeds in real time, using OpenCV for face recognition of the speaker and the Google Cloud Speech-to-Text API for speech analysis.

[1278] An emotion recognition engine (e.g., IBM Watson Emotion Recognition) is used to analyze the user's emotions and generate appropriate responses and comments.

[1279] Device:

[1280] Display received responses and comments to the user.

[1281] Suggestions and measures based on sentiment analysis are displayed to the user.

[1282] 3. Meeting Management Support:

[1283] server:

[1284] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or time is running out.

[1285] Device:

[1286] Display reminder messages and suggestions to the user and send the user's responses and actions to the server.

[1287] 4. Meeting logging and feedback:

[1288] server:

[1289] The server records the meeting log and stores the content of the discussion and the progress of the agenda in a database.

[1290] After the meeting, the generated logs and analysis reports are provided to the user so that they can be used in future meetings.

[1291] Device:

[1292] Users can review meeting recordings and logs and edit them as needed.

[1293] Users can provide feedback and rate the AI's performance to identify areas for improvement in future meetings.

[1294] Specific examples

[1295] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[1296] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, improvements can be made to future meetings.

[1297] Prompt Sentence Examples

[1298] Here are some example prompts to input to the generative AI model:

[1299] I have a question about the functionality of the "Video Chat AI" system I will be using in my next online meeting. Could you please explain in detail how this system, which is equipped with an emotion recognition engine, analyzes the user's emotions and generates appropriate responses and comments? Also, could you please explain in detail how it will support the progress of the meeting?

[1300] This system ensures smooth progress of meetings and manages participants' emotions.

[1301] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1302] Step 1: Prepare for the meeting

[1303] server:

[1304] The server receives advance notices of upcoming meetings. As input, meeting schedule information is provided, e.g., data retrieved from the Google Calendar API.

[1305] The server generates a participant list based on this input data and retrieves data related to each participant from a past conference database (e.g., MySQL). The output is a participant list and the results of an analysis of past speaking tendencies and roles.

[1306] Device:

[1307] The device will send the generated participant list and meeting notification to the user via email or in-app notification.

[1308] Specific behavior:

[1309] The attendee list and meeting materials are displayed on the screen for easy user confirmation, and meeting notifications are sent via email and in-app notifications.

[1310] Step 2: Start the meeting and authenticate users

[1311] server:

[1312] The server receives a signal to start a conference, and inputs include a conference start instruction and each user's authentication information.

[1313] The server connects to a SaaS (Software as a Service) service (e.g., Zoom, Microsoft Teams) to start the conference. The output is the conference room URL and the user authentication result.

[1314] Device:

[1315] The terminal starts the video chat software and sends the user's account information to the server, with the input data being the user's authentication information.

[1316] If the authentication information is successful, the terminal displays the conference room URL returned by the server. The output is the result of providing the conference room URL and verifying the settings information.

[1317] User:

[1318] Users receive a URL or code for joining the meeting and join the meeting.

[1319] The user inputs their preferred speech style and AI behavior settings into the interface. The input data is user preference information, which is then sent to the server.

[1320] Specific behavior:

[1321] The server processes the authentication information and, if successful, provides the device with a meeting room URL, which the user can use to join the meeting in real time.

[1322] Step 3: Analyze the video and audio feeds

[1323] server:

[1324] The server analyzes the video and audio feeds in real time. The input data are the video and audio feeds sent from the devices.

[1325] The speaker's face is recognized using OpenCV, and speech analysis is performed using the Google Cloud Speech-to-Text API. Emotions are analyzed using an emotion recognition engine (e.g., IBM Watson Emotion Recognition). The output is the speaker's face position, speech analysis results, and emotion analysis results.

[1326] Device:

[1327] Displays the backchannels and comments received from the server. The input data is the analysis results sent from the server. The output is the display of feedback to the user.

[1328] Specific behavior:

[1329] When the user starts speaking, the device sends video and audio data to the server, which returns the analysis results to the device and displays them to the user.

[1330] Step 4: Sentiment analysis and response

[1331] server:

[1332] An emotion recognition engine is used to analyze the user's emotions, and the input data is video and audio feed from the device.

[1333] Based on the results of the sentiment analysis, appropriate responses and comments are generated and sent to the device. The output is the sentiment analysis results and the comments and responses based on them.

[1334] Device:

[1335] The received comments and responses are displayed to the user. The input data is the emotion analysis results sent from the server. The output is the content displayed to the user.

[1336] Specific behavior:

[1337] If the user is nervous, display a pop-up notification such as "Would you like to take a short break?"

[1338] Step 5: Managing the meeting

[1339] server:

[1340] The server monitors the progress of the conference, and input data includes the progress status and time information of the conference.

[1341] If the agenda is not progressing or time is running out, a reminder is generated and sent to the terminal. The output is the reminder message.

[1342] Device:

[1343] Display reminders received from the server to the user. The input data is the reminder message sent from the server. The output is the notification content to the user.

[1344] Specific behavior:

[1345] When there is little time left, a notification such as "10 minutes remaining" will be displayed.

[1346] Step 6: Meeting logging and feedback

[1347] server:

[1348] The server records the conference log, and input data includes the content of the conference and its progress.

[1349] The system stores the content of comments and the progress of the agenda in a database, and provides the user with logs and analysis reports for use in future meetings. The output is a meeting log and analysis report.

[1350] Device:

[1351] The conference log and analysis report received from the server are displayed so that the user can check them. The input data are the conference log and analysis report. The output is the content displayed to the user.

[1352] It provides an interface for users to input feedback and evaluate the AI's performance. The input is the user's feedback. The output is the feedback result.

[1353] Specific behavior:

[1354] After the meeting, the recording and log of the meeting are provided to the user, who can review them and provide feedback. This will help evaluate the AI's performance and help improve it for future meetings.

[1355] (Application example 2)

[1356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1357] Analyzing employee conversations and emotions in real time and providing appropriate feedback during factory meetings and conferences is a difficult task. It is particularly important to ensure that meetings proceed smoothly and create an environment where employees can participate without feeling nervous or stressed. It is also necessary to efficiently manage meeting logs and recordings and analyze the information for use in future meetings.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1359] In this invention, the server includes: means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referencing a past conference database; means for analyzing video and audio feeds to recognize the speaker's face and perform voice analysis; means for generating appropriate backchannels and comments based on the content of the speech and presenting them to the user in real time; and means for analyzing user emotions in real time using an emotion recognition engine and generating proposals and countermeasures based on the emotion analysis results. This enables real-time and appropriate feedback and emotion management in factory conferences and meetings. It also enables smooth progress in meetings and provides an environment where employees can participate with peace of mind. Furthermore, the recorded meeting logs and recordings can be used to improve future meetings.

[1360] A "participant list" is a list of people participating in a conference or meeting, and includes information for understanding the role and past speaking tendencies of each participant.

[1361] A "conference database" is a collection of information that stores data such as logs, audio recordings, and video recordings of past meetings, and is used to analyze the speaking tendencies and roles of each participant.

[1362] "Speech tendency" refers to a pattern that indicates what each participant talks about and how often they speak in a conference or meeting.

[1363] "Roles" indicate the position and scope of responsibility of each participant in a conference or meeting, and include, for example, leader, administrative staff, technical staff, etc.

[1364] A "video feed" is a stream of video data that broadcasts video of conference or meeting participants in real time.

[1365] An "audio feed" is a stream of audio data that delivers the voices of conference or meeting participants in real time.

[1366] "Facial recognition" is a technology that analyzes video data to identify specific people, and is used to identify speakers.

[1367] "Voice analysis" is a technology that analyzes voice data to understand what is being said, and is used to generate appropriate responses and comments.

[1368] An "emotion recognition engine" is software or hardware for analyzing a user's emotional state from facial expressions, voice, etc.

[1369] "Ai-chi" is a short phrase or expression used during a conversation to show understanding and empathy for what the other person is saying, and is used to facilitate communication.

[1370] A "comment" is a word used to express an appropriate opinion or reaction to something said during a conference or meeting.

[1371] A "proposal" is a specific idea or action point that encourages new opinions or actions based on what was said during a conference or meeting.

[1372] "Countermeasures" are specific actions or methods proposed to solve the problem based on the content of the comments and the results of emotion analysis.

[1373] A "log" is data that records what is said during a conference or meeting, the progress of the agenda, etc.

[1374] "Recording" refers to data that records video and audio of a conference or meeting.

[1375] An "analysis report" is a document that evaluates the effectiveness and progress of a meeting based on recorded logs and video data, and summarizes the analysis results to be used in future meetings.

[1376] "Reminds" are notifications and alerts that monitor the progress of meetings and conferences and prompt necessary action if things are not going as planned or time is running out.

[1377] This invention provides a "communication support robot system" for smoothly conducting conferences and meetings within a factory. How the system is implemented will be explained below in detail.

[1378] System Overview

[1379] This system mainly consists of a server, robot terminals, and users. The server is responsible for storing and analyzing meeting data and for real-time communication. The robot terminals are devices that allow direct interaction with users within the factory and are equipped with sensors such as cameras and microphones.

[1380] Server Features

[1381] 1. Participant list acquisition and conference database reference function:

[1382] The server receives advance notice of the meeting and a list of participants, and analyzes each participant's speaking tendencies and roles by referring to a database of past meetings.

[1383] 2. Video and audio feed analysis:

[1384] The server analyzes the video and audio feeds sent from the robot terminal in real time, performs facial recognition and voice analysis of the speaker, and analyzes the user's emotions using an emotion recognition engine (e.g., Microsoft's Azure Cognitive Services).

[1385] 3. Sentiment analysis-based suggestion and countermeasure generation function:

[1386] The server generates appropriate responses and comments based on the analysis results, and if necessary, suggests ways to relax. This data is then sent to the robot terminal in real time.

[1387] 4. Meeting progress support function:

[1388] The server monitors the progress of the meeting and generates reminders if the meeting is not progressing as planned or if time is running out.

[1389] 5. Meeting logging and analysis report generation function:

[1390] The server records the meeting log, saving what was said and the progress of the agenda, and generates an analytical report for later review.

[1391] Robot terminal functions

[1392] 1. User input and authentication:

[1393] When the meeting starts, the robot terminal launches the video chat software, the user enters account information into the interface, and the authentication information is sent to the server.

[1394] 2. Get real-time feeds:

[1395] The robot terminal uses a camera and microphone to capture the user's video and audio feeds in real time and transmits them to a server.

[1396] 3. Display of suggestions and measures:

[1397] The system displays to the user the responses, comments, and suggestions and measures based on sentiment analysis sent from the server, and receives the user's responses.

[1398] 4. Processing example:

[1399] For example, if a leader is detected to be nervous during a presentation at a quality control meeting in a factory, the robot will respond by saying, "It's okay, stay calm. What I've just explained is very important."

[1400] The robot will also detect if the agenda is not progressing during a meeting and display a reminder, saying, "We need to confirm the next step on this topic."

[1401] Prompt Sentence Examples

[1402] "If you sense tension in a meeting, a prompt will suggest:

[1403] Generate suggestions to ease the leader's tension by saying, "It's okay to stay calm. What I just explained is very important."

[1404] In this way, the communication support robot system can smoothly conduct conferences and meetings within factories and provide real-time support for employee emotional management, thereby improving meeting efficiency and reducing stress for participants.

[1405] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1406] Step 1:

[1407] Obtaining and pre-analyzing the participant list

[1408] Before the meeting starts, the server receives advance notice of the meeting, obtains the participant list, and then analyzes each participant's speaking habits and roles by referring to a database of past meetings.

[1409] Input: Meeting advance notices, participant list, past meeting database

[1410] Data processing and calculation: The server reads the conference database and compares it with the participant list, analyzing the frequency of each participant's comments and topic trends.

[1411] Output: Analysis results (commentary trends, roles)

[1412] Step 2:

[1413] Conference start and user authentication

[1414] When the conference starts, the terminal launches the video chat software, prompts the user to enter account information, and transmits the authentication information to the server.

[1415] Input: User account information, authentication information

[1416] Data processing and data calculation: The terminal sends the entered account information to the server, and the server verifies the authentication information and confirms the user.

[1417] Output: Authentication result (authentication success / failure)

[1418] Step 3:

[1419] Acquiring and sending video and audio feeds

[1420] The device uses a camera and microphone to capture the user's video and audio feeds in real time during the meeting and transmits them to the server.

[1421] Input: Video feed, Audio feed

[1422] Data processing and data calculation: The device sends the acquired video and audio feeds to the server.

[1423] Output: Video feed and audio feed sent

[1424] Step 4:

[1425] Real-time video and audio analysis

[1426] The server analyzes the received video and audio feeds in real time, performs face recognition and voice analysis of the speaker, and also analyzes the user's emotions using an emotion recognition engine.

[1427] Input: Video feed, Audio feed

[1428] Data processing and data calculation: The server uses a facial recognition algorithm and a voice analysis engine to identify the speaker, and an emotion recognition engine to analyze the user's emotions.

[1429] Output: Face recognition results, voice analysis results, emotion analysis results

[1430] Step 5:

[1431] Generate and present backchannels and comments

[1432] The server generates appropriate responses and comments based on the content of the speech and the results of emotion analysis, and sends them to the device, which then displays them to the user and outputs them as audio.

[1433] Input: Face recognition results, voice analysis results, emotion analysis results

[1434] Data processing and data calculation: The server uses the generative AI model to generate appropriate responses and comments, and sends them to the device.

[1435] Output: backchannel, comments

[1436] Step 6:

[1437] Meeting progress monitoring and reminder generation

[1438] The server monitors the progress of the meeting in real time, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which then notifies the user.

[1439] Input: Meeting progress (log, timestamp)

[1440] Data processing and data calculation: The server analyzes the progress, generates reminders and sends them to the device.

[1441] Output: Reminder notification

[1442] Step 7:

[1443] Recommendation generation based on emotion changes

[1444] If the user's emotional state changes to a negative state, the server generates relaxation suggestions and sends them to the device, which displays them to the user and also outputs them audibly.

[1445] Input: Sentiment analysis results

[1446] Data processing and data calculation: The server uses the generative AI model to generate relaxation suggestions and send them to the device.

[1447] Output: Proposal notification

[1448] Step 8:

[1449] Meeting logging and analytical report generation

[1450] The server records the meeting log, saving the content of comments and the progress of the agenda. After the meeting, an analysis report is generated based on this data, allowing users to use the results in future meetings.

[1451] Input: Meeting log (contents, progress, timestamp)

[1452] Data processing and data calculation: The server stores the log data and generates an analysis report after completion.

[1453] Output: Log data, analysis report

[1454] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1455] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1456] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1457] [Fourth embodiment]

[1458] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1459] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1460] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1461] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1462] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1463] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1464] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1465] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1466] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1467] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1468] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1469] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1470] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1471] This invention provides a system called "Video Chat AI" that facilitates smooth communication during online meetings. This system links multiple functions on the server and terminal sides to efficiently and effectively conduct meetings.

[1472] System Overview

[1473] server:

[1474] The server receives advance notification of the meeting and a list of participants, and refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes the video and audio feeds during the meeting in real time, performing facial recognition and audio analysis of the speakers. It generates backchannels and comments in real time and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[1475] Device:

[1476] The device displays the user's responses and comments received from the server in real time, sends the user's responses and operations to the server, and records the meeting log, providing a saving and editing function so that the user can check it after the meeting has ended.

[1477] A natural language explanation of the program's processing

[1478] 1. User Awareness and Initial Setup:

[1479] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device side, the video chat software launches when the meeting starts, and the user enters their account information and sends authentication information to the server.

[1480] 2. Analyzing video and audio feeds:

[1481] The server analyzes the video and audio feeds in real time, performs facial recognition and speech analysis of the speaker, generates appropriate responses and comments, and sends them to the device, where they are displayed to the user and receive their responses.

[1482] 3. Meeting Management Support:

[1483] The server monitors the progress of the meeting, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which displays it to the user so that the user can smoothly proceed with the meeting.

[1484] 4. Meeting logging and feedback:

[1485] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[1486] Specific examples

[1487] For example, consider the case where User A participates in an online conference. The server analyzes User A's past speaking habits in advance, and generates appropriate backchannels and comments while analyzing User A's comments in real time during the conference. As User A continues to speak, backchannels sent from the server (e.g., "I see, that's interesting") are displayed on the device, smoothly advancing the conversation. Furthermore, as the conference draws to a close, the server generates a reminder saying, "There are 10 minutes left," and the device notifies User A of this, allowing the conference to proceed efficiently.

[1488] This allows users to maintain natural communication during online meetings and ensure important points are covered. Furthermore, after the meeting, users can identify areas for improvement for the next meeting through log recording and feedback, enabling continuous improvement.

[1489] The processing flow will be explained below.

[1490] Step 1:

[1491] server:

[1492] Receive advance notice of meetings, obtain a list of participants, and analyze each participant's speaking habits and roles by referencing a database of past meetings.

[1493] Device:

[1494] When the meeting begins, the video chat software launches, the user enters their account information, and the authentication information is sent to the server.

[1495] User:

[1496] Users input their speaking preferences and AI behavior settings into the interface.

[1497] Step 2:

[1498] server:

[1499] It analyzes video and audio feeds in real time, performs facial recognition and audio analysis of the speaker, and generates appropriate responses and comments.

[1500] Device:

[1501] Displays the user any backchannels or comments received from the server. Sends the user's responses and actions to the server.

[1502] User:

[1503] The user can review the AI's suggestions and choose to accept or ignore them. The AI ​​responds with feedback and additional information in real time while the user is speaking.

[1504] Step 3:

[1505] server:

[1506] Monitor meeting progress and generate reminders if the agenda is falling behind or if the meeting is running out of time.

[1507] Device:

[1508] Display a reminder message to the user. Send the user's response or operation to the server.

[1509] User:

[1510] The user checks the reminder message and adjusts the progress of the meeting as necessary.

[1511] Step 4:

[1512] server:

[1513] Record a meeting log, saving what was said and what was discussed, and generating an analytical report after the meeting.

[1514] Device:

[1515] After the meeting, the recorded log and generated analysis report are provided to the user, and the user can edit and save the log as needed.

[1516] User:

[1517] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance.

[1518] Step 5:

[1519] server:

[1520] Receive feedback from users and make adjustments to be utilized in future meetings.

[1521] Device:

[1522] Optimize settings for your next meeting based on accumulated data and user feedback.

[1523] User:

[1524] Users can use the saved meeting logs and analysis data to prepare for their next meeting.

[1525] Example 1

[1526] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1527] In online meetings, communication between participants can sometimes be disrupted. Participants often lose their ability to speak, or the agenda often doesn't move forward. Furthermore, recording and post-meeting analysis of meeting content is often inadequate, resulting in a lack of feedback to improve future meetings. A solution to these problems is needed.

[1528] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1529] In this invention, the server includes a means for acquiring identification information for the conference and analyzing the behavioral tendencies and roles of each participant by referring to past information records, a means for analyzing image and audio feeds to perform face recognition and acoustic analysis of speakers, a means for generating appropriate reactions and comments based on the content of comments and presenting them to participants in real time, a means for monitoring the progress of the conference and issuing warnings if the agenda is not progressing or time is running out, a means for saving the conference record and generating analysis results to utilize the content of comments and the progress of the agenda for future conferences, and a function for transmitting the generated reactions and comments to terminals and presenting them to conference participants in real time. This enables smooth communication in online conferences and efficient recording and analysis of conference content.

[1530] "Conference identification information" is data that identifies a particular conference and includes information about participants and agenda items.

[1531] "Past information records" are data that record the content of statements and progress of previous meetings.

[1532] "Means for analyzing behavioral tendencies and roles" is a function that analyzes the characteristics and roles of each participant based on their past comments and participation status.

[1533] An "image feed" is video data transmitted in real time from a video device such as a camera.

[1534] An "acoustic feed" is audio data transmitted in real time from an audio device such as a microphone.

[1535] "Speaker facial recognition" is a technology that analyzes video data to identify specific people.

[1536] The means of "acoustic analysis" is a function that analyzes audio data to understand the speaker's voice and content.

[1537] "Means for generating reactions and mentions" is a function that automatically creates responses and comments based on the analyzed content of statements.

[1538] "Means for monitoring the progress of a meeting" refers to a function that monitors the progress of agenda items and the allocation of speaking time in real time.

[1539] The "means of issuing a warning" is a function that notifies participants when the meeting is not progressing or when time is running out.

[1540] "Means for saving meeting records" refers to a function that records the contents of a meeting in real time and saves them for later reference.

[1541] The "means for generating analysis results" is a function that analyzes recorded meeting data and provides useful feedback for future meetings.

[1542] "Having the ability to present in real time" refers to technology that allows participants to instantly see their reactions and comments.

[1543] The present invention provides a system for facilitating communication during online conferences. In this system, a server and terminals work in cooperation with each other to support the progress of the conference. Specific embodiments of the system are described below.

[1544] Server Operation

[1545] The server receives advance notice of the meeting, retrieves the list of participants, and analyzes the behavioral trends and roles of each participant by referencing past information records. For this purpose, a database search engine or machine learning model can be used.

[1546] After the meeting starts, the server receives and analyzes the image and audio feeds in real time. It uses facial recognition and acoustic analysis technologies to identify the speaker and analyze the content of their speech. Specifically, OpenCV is used for facial recognition and IBM Watson Speech to Text API is used for audio analysis.

[1547] The server generates appropriate responses and comments based on the content of the comment and sends them to the device in real time using a generative AI model (e.g., GPT-3).

[1548] In addition, the server monitors the progress of the meeting and generates alerts when the agenda is not progressing or time is running out, which are also sent to the terminal in real time and notified to the user.

[1549] After the meeting ends, the server saves the meeting record and generates analysis results based on the content of the comments and the progress of the agenda. These analysis results are provided to the user for use in future meetings.

[1550] Device behavior

[1551] The terminal receives input from the user and sends it to the server. When a conference starts, the terminal launches the video chat software, prompts the user to enter their account information, and sends the authentication information to the server. After authentication, the conference interface is displayed.

[1552] During the meeting, real-time reactions and comments sent from the server are displayed, allowing users to maintain natural conversations. Alerts from the server are also displayed to support the progress of the meeting.

[1553] After the meeting, the device provides the user with a recorded meeting log and recording, which the user can review and edit as necessary.

[1554] Specific examples

[1555] When User A participates in an online conference, the server analyzes User A's past speaking habits in advance and generates appropriate responses and comments. For example, when User A makes a comment during a conference, the server generates a response such as "That's interesting. Please tell us more about it," and sends it to the device. The device displays this to User A in real time, smoothly advancing the conversation.

[1556] As the meeting draws to a close, the server generates a warning such as "10 minutes remaining," and the terminal notifies User A of this, thereby supporting time management.

[1557] This allows User A to maintain natural communication during online meetings and efficiently progress important agenda items. In addition, the logs and feedback recorded after the meeting allow User A to identify areas for improvement in the next meeting.

[1558] Example prompts for generative AI models

[1559] Here are some example prompts you can enter into your generative AI model:

[1560] In an online conferencing system using "video chat AI," the server identifies the speaker in real time during the meeting and generates appropriate responses and comments. Please explain the specific operations at each step of this system, including the user authentication process and the specific processing of real-time analysis during the meeting.

[1561] Using this prompt, the generative AI model can provide detailed explanations and concrete examples.

[1562] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1563] The flow of this system's program processing

[1564] Step 1: User authentication

[1565] Step 2: Prepare for the meeting

[1566] Step 3: Real-time analytics during the meeting

[1567] Step 4: Meeting facilitating support

[1568] Step 5: Record the meeting and provide feedback

[1569] Specific explanation of each processing step

[1570] Step 1: User authentication

[1571] Input: User account information (email address, password)

[1572] Specific behavior:

[1573] The server waits for the user to launch the video chat software and enter their account information. When the user enters their email address and password, the device sends this to the server, which then compares the received information with a database of past authentications and authenticates the user.

[1574] Data processing and calculation:

[1575] The server hashes the entered account information and compares it with an existing database. If authentication is successful, the server sends an authentication success message to the terminal.

[1576] Output: Authentication result (success / failure), if successful, the conference interface is displayed

[1577] Step 2: Prepare for the meeting

[1578] Input: Participant list, past conference database

[1579] Specific behavior:

[1580] The server receives advance notice of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze the behavioral trends and roles of each participant. The user checks the start time of the meeting and makes any necessary preparations. The server then sends a reminder before the meeting.

[1581] Data processing and calculation:

[1582] Based on the participant list, past meeting data is extracted from the database, and an analytical algorithm is used to analyze behavioral trends and roles. The analysis results are stored on the server and used for real-time analysis during the meeting.

[1583] Output: Participant behavioral trends and roles, reminder notifications

[1584] Step 3: Real-time analytics during the meeting

[1585] Input: Video feed, audio feed

[1586] Specific behavior:

[1587] When a meeting begins, the server acquires and analyzes video and audio feeds in real time. It performs facial recognition and audio analysis of the speaker, generates appropriate responses and comments, and sends them to the device. The user can view these on the screen. Specifically, it uses OpenCV for facial recognition and IBM Watson Speech to Text API for audio analysis.

[1588] Data processing and calculation:

[1589] Facial recognition algorithms are used to identify speakers from video feeds, and speech recognition technology is used to convert speech into text from audio feeds, which is then used to generate real-time responses and comments using generative AI models (e.g., GPT-3).

[1590] Output: Real-time reactions and comments (sent to terminal)

[1591] Step 4: Meeting facilitating support

[1592] Input: Meeting progress, time management data

[1593] Specific behavior:

[1594] The server monitors the progress of the meeting and generates a warning if the agenda is not progressing or if time is running out. This information is sent to the terminal in real time and notifies the user.

[1595] Data processing and calculation:

[1596] It analyzes progress and time management data in real time and generates alerts based on set conditions, such as when no progress is made on an agenda item for a certain period of time or when a meeting is 10 minutes away from the scheduled end time.

[1597] Output: Warning message (sent to terminal)

[1598] Step 5: Record the meeting and provide feedback

[1599] Input: Meeting video feed, audio feed, speech log

[1600] Specific behavior:

[1601] The server stores the meeting records and generates analysis results based on the content of comments and the progress of the agenda. After the meeting, users can check the logs and recordings and make any necessary edits.

[1602] Data processing and calculation:

[1603] It stores the video and audio feeds of the meeting and analyzes speech logs to generate insights that can be used for future meetings.

[1604] Output: Meeting logs, recordings, and analysis results (provided to users)

[1605] (Application example 1)

[1606] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1607] With conventional online conference systems and virtual store systems, it was difficult to maintain smooth communication with participants and customers. In particular, important topics could not be advanced smoothly during the meeting or customer service process, leaving participants and customers frustrated. This led to a decline in meeting efficiency and customer service quality, ultimately resulting in a deterioration in overall business performance.

[1608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1609] In this invention, the server includes means for acquiring a participant list and analyzing the speaking tendencies and roles of each participant by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the face and perform audio analysis of the speaker, means for generating appropriate interjections and comments according to the content of the speech and presenting them to the user in real time, and means for analyzing the facial expressions and speech content of customers in real time and providing comments and advice on appropriate ways to respond, thereby enabling the smooth progress of the conference and customer service and smooth communication that satisfies users and customers.

[1610] A "participant list" is a list containing information about people who will be participating in a conference.

[1611] The "past conference database" is a collection of information that stores records and details of conferences that have been held in the past.

[1612] "Speech trends" indicate the characteristics of the content and patterns of speech that a particular participant often makes during a meeting.

[1613] "Roles" refer to the duties and responsibilities of each participant in a meeting.

[1614] A "video feed" is a continuous stream of video data captured through a camera.

[1615] An "audio feed" is a continuous stream of audio data captured through a microphone.

[1616] "Facial recognition" is a technology that analyzes images captured by a camera to identify the face of a specific individual.

[1617] "Voice analysis" is a technology that analyzes the content of speech and the characteristics of the speaker based on acquired voice data.

[1618] "Ai-chi" refers to a reaction or response that shows that you are listening.

[1619] A "comment" is an opinion or additional information about a particular statement.

[1620] "Customer" means a consumer who visits the virtual store to purchase products or services.

[1621] "Facial expressions" are the movements and changes that appear on a person's face to show their emotions.

[1622] "Utterance content" refers to the specific content that is spoken in a conversation or statement.

[1623] "Response methods" refer to the specific means or methods of responding to customers or participants.

[1624] "Comments and Advice" means advice or opinions given based on statements or circumstances.

[1625] "Customer service" refers to the act of explaining and selling products and services to customers.

[1626] A "remind" is a notification or instruction given to remind you of a specific thing.

[1627] "Meeting progress" indicates the progress of the meeting agenda or topic.

[1628] "Logging" means recording the details of what is said and what happens during a meeting or customer service session.

[1629] An "analysis report" is a report summarizing the results of an analysis based on recorded data.

[1630] This invention provides a system that enables smooth communication in virtual stores and online meetings. Specifically, the server and terminals work together to analyze the facial expressions and speech of customers and participants in real time, and provide appropriate responses and advice.

[1631] Hardware and software used

[1632] 1. Hardware:

[1633] Smart glasses (e.g., Google Glass, Microsoft HoloLens)

[1634] Server (e.g. AWS, Google Cloud Platform)

[1635] 2. Software:

[1636] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1637] Facial recognition API (e.g. Microsoft Azure Face API)

[1638] Real-time communication libraries (e.g., WebRTC)

[1639] System Configuration

[1640] The server has the following means:

[1641] 1. A means of obtaining a list of participants and analyzing each participant's speaking tendencies and role by referencing a database of past conferences.

[1642] 2. A means of analyzing the video and audio feeds to perform facial recognition and voice analysis of the speaker.

[1643] 3. A means of generating appropriate responses and comments based on what is being said and presenting them to the user in real time.

[1644] 4. A means of analyzing customers' facial expressions and speech in real time and providing comments and advice on how to respond appropriately.

[1645] The terminal has the following means:

[1646] 1. A means of displaying backchannels and comments received from the server and sending user responses and actions to the server.

[1647] 2. A means of recording past customer service data and providing the ability to review it later.

[1648] Program processing explanation

[1649] The server receives the login information sent by the user from the smart glasses, analyzes the customer's speech patterns and facial expression patterns by referring to past customer service data, analyzes real-time video and audio feeds, and uses facial expression and voice recognition APIs to understand the customer's emotions and speech content. It then generates appropriate responses and advice and sends them to the smart glasses.

[1650] The smart glasses display the responses and advice sent from the server on the device. As the user continues the conversation with the customer, they respond according to the information displayed on the smart glasses. After the conversation is over, the acquired data is sent back to the server and a feedback report is generated for the next time.

[1651] Specific examples

[1652] For example, if staff member A wears smart glasses while serving customers in a virtual store, the server will analyze customer B's purchase history and preferences based on past data. When customer B visits the store, the smart glasses will display advice such as, "Please recommend new products related to products that customer B previously purchased." If customer B asks a question or looks gloomy, the server will analyze that information and provide staff member A with a comment such as, "The customer seems anxious. Please provide a reassuring response."

[1653] Prompt Sentence Examples

[1654] Analysis of customer conversations in a virtual store:

[1655] Perform facial recognition and sentiment analysis on Customer B to generate an appropriate response in real time.

[1656] If Customer B asks, "Can you tell me more about this product?" you respond with, "Of course. The main features of this product are..."

[1657] Store associates use smart glasses to view advice and reminders and help communicate with customers.

[1658] This allows virtual store staff to provide appropriate customer service in real time, improving customer satisfaction.

[1659] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1660] Step 1:

[1661] The server retrieves the participant list and analyzes each participant's speaking tendencies and roles by referencing a database of past conferences. The server receives the participant list and past conference data as input, performs statistical analysis based on which patterns for each participant are extracted, and generates speaking tendencies data and role information for each participant as output.

[1662] Step 2:

[1663] The server collects video and audio feeds and analyzes them in real time. As input, the video and audio feeds sent from the smart glasses are provided. These feeds are analyzed using facial expression recognition API and speech recognition API to perform facial recognition and speech analysis. As output, analyzed facial expression data and speech content data are obtained.

[1664] Step 3:

[1665] The server generates appropriate backchannels and comments based on the content of the speech. The speech content data obtained in step 2 is used as input. Based on this, appropriate backchannels and comments are generated using natural language processing (NLP) technology. The output is generated as an appropriate response or comment to the user.

[1666] Step 4:

[1667] The server analyzes the customer's facial expressions and speech content in real time, and generates comments and advice indicating the appropriate response method. The facial expression data and speech content data acquired in step 2 are used as input. Based on the analysis results, the server understands the customer's emotional state and requests, and uses a generative AI model to present the appropriate response method. Specific comments and advice are generated as output.

[1668] Step 5:

[1669] The device displays the backchannels, comments, and advice received from the server on the smart glasses. The backchannels, comments, and advice data sent from the server are used as input. This is displayed in real time on the smart glasses' display and provided to the user. The output is the information that the user actually views.

[1670] Step 6:

[1671] The terminal records past customer service data and generates a feedback report for the next time. Input includes user responses and conversations with customers. This data is recorded as a log and sent to the server. The server analyzes the data and compiles it into a feedback report to be used in the next customer service. As output, a feedback report is generated and provided to the user.

[1672] Through these steps, a system will be established that provides appropriate support for customer service in virtual stores in real time, improving customer satisfaction.

[1673] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1674] This invention provides a "Video Chat AI" system for smooth communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions on the server side and the terminal side to efficiently and effectively conduct meetings.

[1675] System Overview

[1676] server:

[1677] The server receives advance notification of the meeting and a list of participants, and then refers to a database of past meetings to analyze each participant's speaking habits and role. It also analyzes video and audio feeds in real time, performing facial and audio analysis of speakers. It also analyzes the user's emotions using an emotion recognition engine, generates appropriate responses and comments, and sends them to the device. It also monitors the progress of the meeting and sends reminders to help ensure the meeting runs smoothly.

[1678] Device:

[1679] The device displays the user's responses and comments received from the server in real time, and sends the user's responses and actions to the server. It also displays suggestions and measures based on the results of user emotion analysis, and manages the atmosphere of the meeting. It also records the meeting log and provides saving and editing functions so that the user can review it after the meeting has ended.

[1680] A natural language explanation of the program's processing

[1681] 1. User Awareness and Initial Setup:

[1682] The server receives advance notification of the meeting and retrieves the participant list. It then refers to a database of past meetings to analyze each participant's speaking habits and role. On the device, the video chat software launches when the meeting begins, and the user enters their account information and sends authentication information to the server. The user then inputs their speaking preferences and AI behavior settings into the interface.

[1683] 2. Analyzing video and audio feeds:

[1684] The server analyzes the video and audio feeds in real time, recognizing the speaker's face and voice. It also uses an emotion recognition engine to analyze the user's emotions and generate appropriate responses and comments. The device displays these responses and comments, along with suggestions and solutions based on the emotion analysis, and receives the user's responses.

[1685] 3. Meeting Management Support:

[1686] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or if time is running out. Furthermore, if the user's emotions change in a negative direction, it generates suggestions and measures to ease the atmosphere. These reminders and suggestions are displayed to the user on the device, and the user's responses and actions are sent to the server.

[1687] 4. Meeting logging and feedback:

[1688] The server records the meeting log and saves the content of comments and the progress of the agenda. After the meeting ends, the generated log and analysis report are provided to the user so that they can be used in future meetings. On the terminal side, the user can check the meeting recording and log and make any necessary edits.

[1689] Specific examples

[1690] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[1691] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, it is possible to identify areas for improvement for future meetings.

[1692] The processing flow will be explained below.

[1693] Step 1:

[1694] server:

[1695] Receive advance notice of upcoming meetings, obtain a list of participants, and analyze each participant's speaking habits and role by referencing a database of past meetings. This allows you to estimate in advance what each participant will contribute and be prepared to provide appropriate support.

[1696] Device:

[1697] When a meeting starts, the video chat software launches. The user enters their account information and sends the authentication information to the server. This allows the software to provide optimal assistance based on each user's settings and past speaking habits.

[1698] User:

[1699] Users input their speaking preferences and AI behavior settings into the interface, such as "don't interrupt me" and "provide timely advice."

[1700] Step 2:

[1701] server:

[1702] The system analyzes video and audio feeds in real time, recognizes the speaker's face and voice, and then uses an emotion recognition engine to analyze the user's emotions. Based on this, appropriate responses and comments are generated and sent to the device. For example, if User A is nervous while speaking, the generated comments might include something like "Relax."

[1703] Device:

[1704] The system displays the backchannels and comments received from the server to the user. It also displays suggestions and measures based on emotion analysis. The system sends the user's responses and actions to the server, enabling real-time feedback.

[1705] User:

[1706] The user can review the AI's suggestions and choose to accept or ignore them. For example, if the AI ​​suggests "Let's move on to the next agenda item," the user can choose to accept or ignore the suggestion.

[1707] Step 3:

[1708] server:

[1709] Monitor the progress of the meeting. Generate reminders if the agenda is not progressing smoothly or if there is little time left in the meeting. Also, if the user's emotions change to a negative direction, generate suggestions or measures to ease the atmosphere. For example, if it feels like the meeting is dragging on, generate a suggestion such as "Let's take a break."

[1710] Device:

[1711] It displays reminder messages and suggestions to users and sends user responses and actions to the server, ensuring smooth progress of the meeting.

[1712] User:

[1713] The user can check the reminder messages and suggestions and adjust the progress of the meeting as necessary, for example, by following the suggestions to "move on to the next agenda item" or "spend more time on the current agenda item."

[1714] Step 4:

[1715] server:

[1716] Record a meeting log, save what was said and the progress of the agenda, and generate an analysis report after the meeting, which can be used for future meetings.

[1717] Device:

[1718] After the meeting, the recorded logs and generated analysis reports are provided to the user. The user can also edit and save the logs as needed, making record management easier.

[1719] User:

[1720] Users can review meeting recordings and logs, make any necessary edits, and provide feedback to evaluate the AI's performance, helping to identify areas for improvement and optimizing the system for future use.

[1721] Step 5:

[1722] server:

[1723] The system receives feedback from users and makes adjustments to be applied to future meetings, thereby enabling continuous improvement of the system.

[1724] Device:

[1725] For your next meeting, we'll optimize your settings based on accumulated data and user feedback, ensuring effective support for future meetings.

[1726] User:

[1727] Users can use the saved meeting logs and analysis data to prepare for their next meeting, thereby improving the efficiency and effectiveness of meetings.

[1728] Example 2

[1729] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] In online meetings, it is difficult to properly manage the emotions of speakers and the progress of the meeting in real time. Particularly when there are many participants, it is necessary to grasp the emotions and progress of each speaker and automatically provide appropriate responses and reminders, but conventional systems have not been able to achieve this. Therefore, improving the efficiency of meetings and ensuring smooth communication between participants is a challenge.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1732] In this invention, the server includes means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referring to a past conference database, means for analyzing the video feed and audio feed to recognize the speaker's face and perform voice analysis, means for analyzing the user's emotions using an emotion recognition engine, generating appropriate backchannels and comments, and presenting them to the user in real time, means for monitoring the progress of the conference and generating reminders when progress on the agenda is not being made or time is running out, and means for recording a conference log and saving the content of comments and the progress of the agenda. This makes it possible to grasp user emotions and the progress of the conference in real time and respond appropriately.

[1733] A "participant list" is a list that includes the names and IDs of all users who will be participating in a conference.

[1734] A "conference database" is a database that stores information about past conferences, and stores data on what was said and the roles of participants.

[1735] "Speech trends" are behavioral patterns obtained by analyzing the content, frequency, and role of each participant in past meetings.

[1736] "Facial recognition" is a technology that analyzes facial images obtained from a video feed and identifies the speaker's face.

[1737] "Speech analysis" is a technology that analyzes audio feeds and converts what is being said into text.

[1738] An "emotion recognition engine" is an engine that analyzes video and audio data to detect a user's emotional state (e.g., joy, sadness, tension, etc.).

[1739] "Responses and comments" refer to feedback and reactions generated based on what is said, and are elements that help smooth the flow of conversation.

[1740] "Reminders" are notifications and alerts about meeting progress and time management, designed to prevent delays and time-outs.

[1741] A "meeting log" refers to a record of all data such as what was said during the meeting, the progress, and the degree to which the agenda was achieved.

[1742] "Suggestions and measures" are specific actions generated based on emotion analysis to improve the user's condition or the progress of the meeting.

[1743] This invention is a "Video Chat AI" system for facilitating communication in online meetings, and in particular includes a form that combines an emotion engine that recognizes the user's emotions. This system links multiple functions of the server and terminals to efficiently and effectively conduct meetings.

[1744] 1. User Awareness and Initial Setup:

[1745] server:

[1746] The server receives advance notice of the meeting and retrieves the attendee list, for example, using the Google Calendar API to retrieve the meeting schedule.

[1747] The server retrieves relevant data from a past conference database (e.g., MySQL) and analyzes participants' speaking tendencies and roles.

[1748] Device:

[1749] When a user joins a meeting, video chat software (e.g., Zoom, Microsoft Teams) is launched.

[1750] The user enters their account information and sends their authentication information to the server to log in.

[1751] Users input their preferred speaking style and AI behavior settings into the interface.

[1752] 2. Analyzing video and audio feeds:

[1753] server:

[1754] The server analyzes the video and audio feeds in real time, using OpenCV for face recognition of the speaker and the Google Cloud Speech-to-Text API for speech analysis.

[1755] An emotion recognition engine (e.g., IBM Watson Emotion Recognition) is used to analyze the user's emotions and generate appropriate responses and comments.

[1756] Device:

[1757] Display received responses and comments to the user.

[1758] Suggestions and measures based on sentiment analysis are displayed to the user.

[1759] 3. Meeting Management Support:

[1760] server:

[1761] The server monitors the progress of the meeting and generates reminders if the agenda is not progressing or time is running out.

[1762] Device:

[1763] Display reminder messages and suggestions to the user and send the user's responses and actions to the server.

[1764] 4. Meeting logging and feedback:

[1765] server:

[1766] The server records the meeting log and stores the content of the discussion and the progress of the agenda in a database.

[1767] After the meeting, the generated logs and analysis reports are provided to the user so that they can be used in future meetings.

[1768] Device:

[1769] Users can review meeting recordings and logs and edit them as needed.

[1770] Users can provide feedback and rate the AI's performance to identify areas for improvement in future meetings.

[1771] Specific examples

[1772] For example, consider the case where user A participates in an online conference. The server analyzes user A's past speaking tendencies in advance, and generates appropriate responses and comments while analyzing what user A says in real time during the conference. Furthermore, an emotion recognition engine is used to analyze user A's emotions from changes in facial expressions and voice, and if necessary, adds emotional nuance to responses such as "I see, that's interesting." If user A appears to be nervous, the server generates a suggestion such as "Shall we take a short break?" and displays this to user A on the device.

[1773] As the meeting draws to a close, the server generates a reminder saying "10 minutes remaining," and the device notifies User A of this, ensuring the meeting proceeds efficiently. After the meeting ends, a recording and log of the meeting are generated, which User A can review, edit, and save as necessary. In addition, by inputting feedback and evaluating the AI's performance, improvements can be made to future meetings.

[1774] Prompt Sentence Examples

[1775] Here are some example prompts to input to the generative AI model:

[1776] I have a question about the functionality of the "Video Chat AI" system I will be using in my next online meeting. Could you please explain in detail how this system, which is equipped with an emotion recognition engine, analyzes the user's emotions and generates appropriate responses and comments? Also, could you please explain in detail how it will support the progress of the meeting?

[1777] This system ensures smooth progress of meetings and manages participants' emotions.

[1778] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1779] Step 1: Prepare for the meeting

[1780] server:

[1781] The server receives advance notices of upcoming meetings. As input, meeting schedule information is provided, e.g., data retrieved from the Google Calendar API.

[1782] The server generates a participant list based on this input data and retrieves data related to each participant from a past conference database (e.g., MySQL). The output is a participant list and the results of an analysis of past speaking tendencies and roles.

[1783] Device:

[1784] The device will send the generated participant list and meeting notification to the user via email or in-app notification.

[1785] Specific behavior:

[1786] The attendee list and meeting materials are displayed on the screen for easy user confirmation, and meeting notifications are sent via email and in-app notifications.

[1787] Step 2: Start the meeting and authenticate users

[1788] server:

[1789] The server receives a signal to start a conference, and inputs include a conference start instruction and each user's authentication information.

[1790] The server connects to a SaaS (Software as a Service) service (e.g., Zoom, Microsoft Teams) to start the conference. The output is the conference room URL and the user authentication result.

[1791] Device:

[1792] The terminal starts the video chat software and sends the user's account information to the server, with the input data being the user's authentication information.

[1793] If the authentication information is successful, the terminal displays the conference room URL returned by the server. The output is the result of providing the conference room URL and verifying the settings information.

[1794] User:

[1795] Users receive a URL or code for joining the meeting and join the meeting.

[1796] The user inputs their preferred speech style and AI behavior settings into the interface. The input data is user preference information, which is then sent to the server.

[1797] Specific behavior:

[1798] The server processes the authentication information and, if successful, provides the device with a meeting room URL, which the user can use to join the meeting in real time.

[1799] Step 3: Analyze the video and audio feeds

[1800] server:

[1801] The server analyzes the video and audio feeds in real time. The input data are the video and audio feeds sent from the devices.

[1802] The speaker's face is recognized using OpenCV, and speech analysis is performed using the Google Cloud Speech-to-Text API. Emotions are analyzed using an emotion recognition engine (e.g., IBM Watson Emotion Recognition). The output is the speaker's face position, speech analysis results, and emotion analysis results.

[1803] Device:

[1804] Displays the backchannels and comments received from the server. The input data is the analysis results sent from the server. The output is the display of feedback to the user.

[1805] Specific behavior:

[1806] When the user starts speaking, the device sends video and audio data to the server, which returns the analysis results to the device and displays them to the user.

[1807] Step 4: Sentiment analysis and response

[1808] server:

[1809] An emotion recognition engine is used to analyze the user's emotions, and the input data is video and audio feed from the device.

[1810] Based on the results of the sentiment analysis, appropriate responses and comments are generated and sent to the device. The output is the sentiment analysis results and the comments and responses based on them.

[1811] Device:

[1812] The received comments and responses are displayed to the user. The input data is the emotion analysis results sent from the server. The output is the content displayed to the user.

[1813] Specific behavior:

[1814] If the user is nervous, display a pop-up notification such as "Would you like to take a short break?"

[1815] Step 5: Managing the meeting

[1816] server:

[1817] The server monitors the progress of the conference, and input data includes the progress status and time information of the conference.

[1818] If the agenda is not progressing or time is running out, a reminder is generated and sent to the terminal. The output is the reminder message.

[1819] Device:

[1820] Display reminders received from the server to the user. The input data is the reminder message sent from the server. The output is the notification content to the user.

[1821] Specific behavior:

[1822] When there is little time left, a notification such as "10 minutes remaining" will be displayed.

[1823] Step 6: Meeting logging and feedback

[1824] server:

[1825] The server records the conference log, and input data includes the content of the conference and its progress.

[1826] The system stores the content of comments and the progress of the agenda in a database, and provides the user with logs and analysis reports for use in future meetings. The output is a meeting log and analysis report.

[1827] Device:

[1828] The conference log and analysis report received from the server are displayed so that the user can check them. The input data are the conference log and analysis report. The output is the content displayed to the user.

[1829] It provides an interface for users to input feedback and evaluate the AI's performance. The input is the user's feedback. The output is the feedback result.

[1830] Specific behavior:

[1831] After the meeting, the recording and log of the meeting are provided to the user, who can review them and provide feedback. This will help evaluate the AI's performance and help improve it for future meetings.

[1832] (Application example 2)

[1833] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1834] Analyzing employee conversations and emotions in real time and providing appropriate feedback during factory meetings and conferences is a difficult task. It is particularly important to ensure that meetings proceed smoothly and create an environment where employees can participate without feeling nervous or stressed. It is also necessary to efficiently manage meeting logs and recordings and analyze the information for use in future meetings.

[1835] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1836] In this invention, the server includes: means for acquiring a participant list and analyzing each participant's speaking tendencies and roles by referencing a past conference database; means for analyzing video and audio feeds to recognize the speaker's face and perform voice analysis; means for generating appropriate backchannels and comments based on the content of the speech and presenting them to the user in real time; and means for analyzing user emotions in real time using an emotion recognition engine and generating proposals and countermeasures based on the emotion analysis results. This enables real-time and appropriate feedback and emotion management in factory conferences and meetings. It also enables smooth progress in meetings and provides an environment where employees can participate with peace of mind. Furthermore, the recorded meeting logs and recordings can be used to improve future meetings.

[1837] A "participant list" is a list of people participating in a conference or meeting, and includes information for understanding the role and past speaking tendencies of each participant.

[1838] A "conference database" is a collection of information that stores data such as logs, audio recordings, and video recordings of past meetings, and is used to analyze the speaking tendencies and roles of each participant.

[1839] "Speech tendency" refers to a pattern that indicates what each participant talks about and how often they speak in a conference or meeting.

[1840] "Roles" indicate the position and scope of responsibility of each participant in a conference or meeting, and include, for example, leader, administrative staff, technical staff, etc.

[1841] A "video feed" is a stream of video data that broadcasts video of conference or meeting participants in real time.

[1842] An "audio feed" is a stream of audio data that delivers the voices of conference or meeting participants in real time.

[1843] "Facial recognition" is a technology that analyzes video data to identify specific people, and is used to identify speakers.

[1844] "Voice analysis" is a technology that analyzes voice data to understand what is being said, and is used to generate appropriate responses and comments.

[1845] An "emotion recognition engine" is software or hardware for analyzing a user's emotional state from facial expressions, voice, etc.

[1846] "Ai-chi" is a short phrase or expression used during a conversation to show understanding and empathy for what the other person is saying, and is used to facilitate communication.

[1847] A "comment" is a word used to express an appropriate opinion or reaction to something said during a conference or meeting.

[1848] A "proposal" is a specific idea or action point that encourages new opinions or actions based on what was said during a conference or meeting.

[1849] "Countermeasures" are specific actions or methods proposed to solve the problem based on the content of the comments and the results of emotion analysis.

[1850] A "log" is data that records what is said during a conference or meeting, the progress of the agenda, etc.

[1851] "Recording" refers to data that records video and audio of a conference or meeting.

[1852] An "analysis report" is a document that evaluates the effectiveness and progress of a meeting based on recorded logs and video data, and summarizes the analysis results to be used in future meetings.

[1853] "Reminds" are notifications and alerts that monitor the progress of meetings and conferences and prompt necessary action if things are not going as planned or time is running out.

[1854] This invention provides a "communication support robot system" for smoothly conducting conferences and meetings within a factory. How the system is implemented will be explained below in detail.

[1855] System Overview

[1856] This system mainly consists of a server, robot terminals, and users. The server is responsible for storing and analyzing meeting data and for real-time communication. The robot terminals are devices that allow direct interaction with users within the factory and are equipped with sensors such as cameras and microphones.

[1857] Server Features

[1858] 1. Participant list acquisition and conference database reference function:

[1859] The server receives advance notice of the meeting and a list of participants, and analyzes each participant's speaking tendencies and roles by referring to a database of past meetings.

[1860] 2. Video and audio feed analysis:

[1861] The server analyzes the video and audio feeds sent from the robot terminal in real time, performs facial recognition and voice analysis of the speaker, and analyzes the user's emotions using an emotion recognition engine (e.g., Microsoft's Azure Cognitive Services).

[1862] 3. Sentiment analysis-based suggestion and countermeasure generation function:

[1863] The server generates appropriate responses and comments based on the analysis results, and if necessary, suggests ways to relax. This data is then sent to the robot terminal in real time.

[1864] 4. Meeting progress support function:

[1865] The server monitors the progress of the meeting and generates reminders if the meeting is not progressing as planned or if time is running out.

[1866] 5. Meeting logging and analysis report generation function:

[1867] The server records the meeting log, saving what was said and the progress of the agenda, and generates an analytical report for later review.

[1868] Robot terminal functions

[1869] 1. User input and authentication:

[1870] When the meeting starts, the robot terminal launches the video chat software, the user enters account information into the interface, and the authentication information is sent to the server.

[1871] 2. Get real-time feeds:

[1872] The robot terminal uses a camera and microphone to capture the user's video and audio feeds in real time and transmits them to a server.

[1873] 3. Display of suggestions and measures:

[1874] The system displays to the user the responses, comments, and suggestions and measures based on sentiment analysis sent from the server, and receives the user's responses.

[1875] 4. Processing example:

[1876] For example, if a leader is detected to be nervous during a presentation at a quality control meeting in a factory, the robot will respond by saying, "It's okay, stay calm. What I've just explained is very important."

[1877] The robot will also detect if the agenda is not progressing during a meeting and display a reminder, saying, "We need to confirm the next step on this topic."

[1878] Prompt Sentence Examples

[1879] "If you sense tension in a meeting, a prompt will suggest:

[1880] Generate suggestions to ease the leader's tension by saying, "It's okay to stay calm. What I just explained is very important."

[1881] In this way, the communication support robot system can smoothly conduct conferences and meetings within factories and provide real-time support for employee emotional management, thereby improving meeting efficiency and reducing stress for participants.

[1882] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1883] Step 1:

[1884] Obtaining and pre-analyzing the participant list

[1885] Before the meeting starts, the server receives advance notice of the meeting, obtains the participant list, and then analyzes each participant's speaking habits and roles by referring to a database of past meetings.

[1886] Input: Meeting advance notices, participant list, past meeting database

[1887] Data processing and calculation: The server reads the conference database and compares it with the participant list, analyzing the frequency of each participant's comments and topic trends.

[1888] Output: Analysis results (commentary trends, roles)

[1889] Step 2:

[1890] Conference start and user authentication

[1891] When the conference starts, the terminal launches the video chat software, prompts the user to enter account information, and transmits the authentication information to the server.

[1892] Input: User account information, authentication information

[1893] Data processing and data calculation: The terminal sends the entered account information to the server, and the server verifies the authentication information and confirms the user.

[1894] Output: Authentication result (authentication success / failure)

[1895] Step 3:

[1896] Acquiring and sending video and audio feeds

[1897] The device uses a camera and microphone to capture the user's video and audio feeds in real time during the meeting and transmits them to the server.

[1898] Input: Video feed, Audio feed

[1899] Data processing and data calculation: The device sends the acquired video and audio feeds to the server.

[1900] Output: Video feed and audio feed sent

[1901] Step 4:

[1902] Real-time video and audio analysis

[1903] The server analyzes the received video and audio feeds in real time, performs face recognition and voice analysis of the speaker, and also analyzes the user's emotions using an emotion recognition engine.

[1904] Input: Video feed, Audio feed

[1905] Data processing and data calculation: The server uses a facial recognition algorithm and a voice analysis engine to identify the speaker, and an emotion recognition engine to analyze the user's emotions.

[1906] Output: Face recognition results, voice analysis results, emotion analysis results

[1907] Step 5:

[1908] Generate and present backchannels and comments

[1909] The server generates appropriate responses and comments based on the content of the speech and the results of emotion analysis, and sends them to the device, which then displays them to the user and outputs them as audio.

[1910] Input: Face recognition results, voice analysis results, emotion analysis results

[1911] Data processing and data calculation: The server uses the generative AI model to generate appropriate responses and comments, and sends them to the device.

[1912] Output: backchannel, comments

[1913] Step 6:

[1914] Meeting progress monitoring and reminder generation

[1915] The server monitors the progress of the meeting in real time, and if the agenda is not progressing or time is running out, it generates a reminder and sends it to the terminal, which then notifies the user.

[1916] Input: Meeting progress (log, timestamp)

[1917] Data processing and data calculation: The server analyzes the progress, generates reminders and sends them to the device.

[1918] Output: Reminder notification

[1919] Step 7:

[1920] Recommendation generation based on emotion changes

[1921] If the user's emotional state changes to a negative state, the server generates relaxation suggestions and sends them to the device, which displays them to the user and also outputs them audibly.

[1922] Input: Sentiment analysis results

[1923] Data processing and data calculation: The server uses the generative AI model to generate relaxation suggestions and send them to the device.

[1924] Output: Proposal notification

[1925] Step 8:

[1926] Meeting logging and analytical report generation

[1927] The server records the meeting log, saving the content of comments and the progress of the agenda. After the meeting, an analysis report is generated based on this data, allowing users to use the results in future meetings.

[1928] Input: Meeting log (contents, progress, timestamp)

[1929] Data processing and data calculation: The server stores the log data and generates an analysis report after completion.

[1930] Output: Log data, analysis report

[1931] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1932] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1933] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1934] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1935] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1936] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1937] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1938] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1939] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1940] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1941] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1942] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1943] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1944] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1945] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1946] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1947] ...

Claims

1. A means to obtain a list of participants and analyze the speaking tendencies and roles of each participant by referencing a database of past conferences. means for analyzing the video and audio feeds to perform facial recognition and audio analysis of the speaker; A means to generate appropriate responses and comments based on the content of the speech and present them to the user in real time; A system including:

2. The system of claim 1 , further comprising means for monitoring the progress of the meeting and providing a reminder when progress on the agenda is not being made or time is running out.

3. 2. The system according to claim 1, further comprising: means for recording a meeting log and saving the contents of comments and the progress of the agenda; and means for generating an analysis report based on the log to be utilized in subsequent meetings.

Images