system

Abstract

We provide the system. [Solution] Receiving means for receiving acoustic information, visual information, bodily movement information, and textual information, A preprocessing means for preprocessing various information acquired by the receiving means, An analysis means for analyzing the information preprocessed by the aforementioned preprocessing means, Integration means for integrating the analysis results obtained by the analysis means, An advice generation means that generates advice based on the analysis results integrated by the aforementioned integration means, An output means for providing the aforementioned advice to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The importance of non-verbal elements in negotiation and dialogue is widely recognized, but the prior art for analyzing them in a multi-faceted and real-time manner has limitations. Specifically, there has been a lack of means to comprehensively understand non-verbal communication elements such as voice tone, facial expressions, and gestures, and to provide effective countermeasures based on the results. In addition, there have been issues regarding the privacy protection of the collected data. As a result, it has been difficult to accurately grasp the negotiation situation and formulate an optimal communication strategy.

Means for Solving the Problems

[0005] This invention provides a system that comprehensively receives, preprocesses, and analyzes acoustic, visual, bodily movement, and textual information. The preprocessed information is analyzed by an analysis means, and the results are integrated by an integration means. Based on the analysis results of the nonverbal elements obtained through this integrated analysis, an advice generation means is activated to generate effective advice. The generated advice is provided to the user visually or audibly. Furthermore, anonymization processing is performed to protect data privacy, creating a secure environment for use. This enables users to receive multifaceted information in real time during negotiations and dialogues, and to achieve effective communication.

[0006] "Acoustic information" refers to information that includes characteristic quantities related to sound, such as the frequency, tone, and volume of speech or sounds.

[0007] "Visual information" refers to image and video data acquired through cameras and sensors, as well as information about the shape and movement of an object derived from them.

[0008] "Physical movement information" refers to data that shows an individual's body movements, gestures, and changes in posture, and is an essential element for motion analysis.

[0009] "Textual information" refers to text data expressed as a string of characters that can be analyzed using natural language processing.

[0010] "Receiving means" refers to a device or module that has the ability to acquire information related to sound, sight, bodily movements, and text from the external environment.

[0011] "Preprocessing means" refers to functions that perform processing such as noise reduction and format conversion in order to format the collected raw data into a state that can be analyzed.

[0012] An "analysis tool" is a module that uses pre-processed data to extract meaningful information and performs calculations to reveal specific patterns or features.

[0013] An "integration tool" is a function that is responsible for the process of centrally combining and evaluating multiple analysis results to deepen overall understanding.

[0014] An "advice generation tool" is an algorithm or module that creates suggestions or recommended actions for a user based on integrated information.

[0015] "Output means" refers to an interface for presenting information or results to the user, and can be expressed visually or audibly.

[0016] "Anonymization" is a technology that removes or transforms personally identifiable information to handle data while protecting privacy. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0019] First, the terms used in the following description will be described.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The multimodal communication analysis AI agent according to the present invention is a system that enables a comprehensive understanding of nonverbal elements in negotiation and dialogue situations by receiving, analyzing, integrating various types of information and providing advice on optimal strategies. Its embodiments are described below.

[0039] First, the user activates the system and collects the necessary data through the terminal. The terminal picks up participants' voices as acoustic information using a microphone and captures video as visual information using a camera. It also obtains physical movement information by detecting the user's movements with sensors. Finally, it obtains text information by inputting the content of negotiations and conversations as text.

[0040] The server receives this information in real time and performs effective preprocessing. Specifically, it removes noise from acoustic data and extracts necessary frames from visual data. It also performs text analysis on text data using natural language processing techniques. Body movement information is converted into a format that can be analyzed by motion recognition algorithms.

[0041] During the analysis phase, the server analyzes each piece of pre-processed information based on its specific characteristics. Voice analysis evaluates participants' emotions and intentions, while facial expression analysis captures subtle facial changes. Gesture analysis tracks hand and body movements and extracts important ones. Text analysis understands the tone and intent of speech to grasp the overall picture.

[0042] Next, the server comprehensively evaluates these analysis results to capture the participants' emotional states and the atmosphere of the negotiation in a unified manner. Based on this integrated analysis, the server generates appropriate advice. The advice generation utilizes deep learning techniques and past pattern analysis results to propose the optimal strategy for the specific situation the user is facing.

[0043] The generated advice is provided to the user through the device. The user receives visual feedback on the device screen and can refer to audio guides. This allows the user to immediately obtain useful information and effectively advance negotiations and discussions.

[0044] Furthermore, the server ensures that all data is anonymized and securely managed to protect user privacy. This allows users to use the system with peace of mind.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The user activates the terminal and starts the multimodal communication analysis system. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting body movement information.

[0048] Step 2:

[0049] The device begins data collection upon receiving a start command from the user. It collects acoustic information using a microphone, acquires visual information using a camera, and records body movement information using sensors. For text information, the user inputs the text into the device.

[0050] Step 3:

[0051] The server receives raw data transmitted from the terminal. This received data includes acoustic, visual, bodily motion, and textual information.

[0052] Step 4:

[0053] The server preprocesses the received data. Noise reduction is performed on acoustic data, and frame division is carried out on visual data. Body movement data is formatted into a unified sensor data format, and text data is converted into a form suitable for language analysis using natural language processing techniques.

[0054] Step 5:

[0055] The server analyzes the pre-processed data. It performs emotion analysis on acoustic information and facial expression analysis on visual information. It performs gesture analysis on bodily movement information and analyzes intent and context on written information.

[0056] Step 6:

[0057] The server integrates the results of each analysis. Based on the integrated information, it evaluates the overall communication situation and understands the participants' emotions and circumstances.

[0058] Step 7:

[0059] The server generates advice from integrated data. It uses deep learning and analysis of historical patterns to suggest optimal negotiation strategies and actions for the user.

[0060] Step 8:

[0061] The server sends the generated advice to the terminal. The terminal presents the results, including the advice, to the user in visual and auditory formats.

[0062] Step 9:

[0063] Users receive advice and analysis results displayed on their devices and decide on their next actions based on them. This enables users to communicate effectively in real time.

[0064] (Example 1)

[0065] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0066] Nonverbal elements play a crucial role in modern negotiations and dialogues. However, comprehensively understanding and effectively analyzing this diverse information is challenging. In particular, there is a need to accurately grasp participants' emotional states and the atmosphere of negotiations in real time and provide appropriate advice based on that understanding, but existing technologies are unable to meet this challenge.

[0067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0068] In this invention, the server includes acquisition means for acquiring audio data, video data, body movement data, and text data; a data processing mechanism for preprocessing the data obtained by the acquisition means; and data analysis means for analyzing the preprocessed data provided by the data processing mechanism. This enables centralized analysis of diverse information and the generation of optimal advice.

[0069] "Acquisition means" refers to a mechanism for collecting audio data, video data, body movement data, and text data and providing them to a server.

[0070] A "data processing mechanism" is a device or method that organizes data obtained through acquisition means and prepares it in a format that can be analyzed.

[0071] A "data analysis tool" is a system or process for analyzing pre-processed data and extracting and evaluating specific information based on its content.

[0072] An "analysis integration tool" is a function that integrates information obtained from individual data analyses and evaluates it from a more holistic perspective.

[0073] A "content generation device" is a mechanism that creates suggestions and advice for users based on integrated analysis results.

[0074] An "information transmission system" is a means of conveying the generated proposal content to users visually or audibly.

[0075] The "evaluation function" is a function within the analysis and integration system that determines the characteristics of emotional states and reactions based on the obtained data.

[0076] "Anonymization" is a technique or method that removes or conceals personally identifiable information from data in order to protect privacy.

[0077] This multimodal communication analysis system is designed to enable users to effectively acquire and analyze information in negotiations and dialogues, and to obtain optimal advice. The system has the functionality to collect and analyze audio data, video data, bodily motion data, and text data.

[0078] The user activates the system and collects necessary data using a terminal. The terminal has a built-in microphone, camera, and motion sensor, and uses this hardware to acquire voice, video, and body movement information. In addition, the content of negotiations and conversations can be entered as text data via a keyboard or other input devices.

[0079] The server receives data from terminals in real time and performs preprocessing using a data processing mechanism. Preprocessing includes noise reduction for audio data, frame extraction for video data, and natural language processing for text data. This prepares the data for analysis.

[0080] The analyzed data is integrated and analyzed using server-generated AI models to understand the emotional state of participants and the atmosphere of negotiations. Based on these analysis results, optimal advice is generated. The advice is provided as the most appropriate response for the current situation, based on a model that has learned past data patterns.

[0081] The device provides feedback to the user with the generated advice. Visual feedback is displayed on the screen, and audio guidance is used as needed. For example, it can show specific negotiation strategies or provide the results of an analysis of the user's emotional state.

[0082] Regarding privacy protection, the server anonymizes all data and securely manages user information. This protection feature allows users to use the system with peace of mind.

[0083] A concrete example would be a situation where, "a client is concerned about the price of a new product, and we need advice on how to emphasize the product's added value instead of lowering the price." An example of a prompt in this case would be:

[0084] Question for the AI ​​agent: If a client expresses concerns about pricing, how can I emphasize the added value of the product without lowering the price?

[0085] In this way, the system provides immediate and effective support for the challenges that users face.

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The user activates the device and collects the data necessary for negotiations and conversations. Inputs include acoustic data acquired by the device's microphone, video data captured by the camera, body movement data detected by sensors, and text data entered by the user. Output is a composite data stream sent to the server.

[0089] Step 2:

[0090] The server receives a composite data stream from the terminal and performs preprocessing using a data processing mechanism. Specific operations include noise reduction on audio data input, extraction of necessary frames from video data, and natural language processing for text data. The output of this processing is a dataset in a format suitable for analysis.

[0091] Step 3:

[0092] The server analyzes the pre-processed data using data analysis tools. Based on each data input, it analyzes emotional nuances from audio data, changes in facial expressions and gestures from video data, and intentions and tone from text data. The output of this step is the analysis results, such as emotional states.

[0093] Step 4:

[0094] The server integrates the analysis results obtained from the data analysis means using the analysis integration means. The analysis results are unified, summarizing the overall atmosphere of the negotiation and the emotional state of the participants. The output of this integration process is the integrated evaluation result.

[0095] Step 5:

[0096] The server generates optimal advice using a generative AI model based on the integrated evaluation results. Based on the integrated results, it devises countermeasures for specific situations the user faces and creates advice statements. The output is the specific advice provided to the user.

[0097] Step 6:

[0098] The terminal provides the user with advice from the server. The advice is displayed visually on the screen and accompanied by specific feedback actions explained by voice guidance. The final output of this step is easy-to-understand advice received by the user.

[0099] Step 7:

[0100] The server anonymizes the acquired data, ensuring secure information management while protecting privacy. Each data element is anonymized and kept inaccessible from the outside. The output of this process is an anonymized, secure data record.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] In many brick-and-mortar stores, sales staff often struggle to accurately understand customers' emotions and intentions, resulting in inadequate customer service. Furthermore, a lack of tools to enable sales staff to analyze customer reactions in real time and propose optimal service approaches poses challenges to improving customer satisfaction and maximizing sales opportunities.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes means for receiving acoustic data, visual data, bodily movement data, and text data; means for preprocessing the various data acquired by the receiving means; and means for analyzing the data preprocessed by the preprocessing means. This makes it possible to analyze emotional state and reaction data in real time during interaction with customers and propose the optimal customer service method.

[0106] "Audio data" refers to audio information acquired via audio input devices such as microphones, and includes audio information related to customer speech and conversations.

[0107] "Visual data" refers to video information acquired by video input devices such as cameras, and includes a series of visual information such as a customer's facial expressions and movements.

[0108] "Body movement data" refers to data that includes information about a person's body movements and posture obtained from acceleration sensors and motion sensors.

[0109] "Text data" refers to data that includes information obtained by converting speech into text information using speech recognition technology, and which includes information that is a transcription of customer utterances.

[0110] "Analysis results" refer to information obtained after analyzing acoustic data, visual data, bodily movement data, and text data, including evaluations of the customer's emotional state and intentions.

[0111] A "strategy generation tool" is a device or algorithm that performs a series of processes to generate an optimal communication strategy based on analysis results.

[0112] A "presentation means" is a means of showing a generated communication strategy to the user, and is a device that transmits information visually or audibly using displays and audio equipment.

[0113] "Optimization suggestion means" refers to a system or process that makes specific suggestions for improving customer service methods based on real-time customer feedback.

[0114] This shows an embodiment for carrying out the invention.

[0115] To realize this application example, the present invention is configured as follows: The server provides an advanced analysis system to support the interaction between the customer and the salesperson. First, acoustic data, visual data, bodily motion data, and text data are acquired using a receiving device. Acoustic data is obtained by recording the customer's voice using a microphone and converting it into text data using speech recognition technology. Visual data is obtained by capturing the customer's facial expressions and gestures with a camera and analyzing it using image recognition technology. Bodily motion data is obtained by detecting the customer's movements with a motion sensor and processing it with a motion analysis algorithm.

[0116] The server processes this data in real time and performs detailed analysis based on the characteristics of each data point using deep learning models. By integrating these analysis results and evaluating customer emotions and intentions, the most effective customer service strategy is generated. The generated strategy is presented to the sales staff and visually fed back using smart glasses or display devices.

[0117] Furthermore, for particularly complex customer requests, adaptive learning of deep learning models can be utilized to improve analysis accuracy. This system also incorporates data anonymization, providing strong protection for customer privacy.

[0118] For example, if analysis of customer voices detects that the customer is excited, the system will advise the salesperson to "explain things carefully to the customer and take your time guiding them." As an example of a prompt, it will generate a question such as, "The customer's tone of voice has started to change. What emotions could this indicate?" to draw the salesperson's attention.

[0119] Thus, the present invention makes it possible to increase customer satisfaction and expand sales opportunities.

[0120] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0121] Step 1:

[0122] The user uses the device's camera and microphone to acquire customer audio data, visual data, body movement data, and conversation data. Audio and visual information are captured as input. This collects basic data on customer interactions.

[0123] Step 2:

[0124] To analyze the received data, the server first removes noise from the audio data and extracts the necessary frames from the visual data. The input here is the original audio and video, and the output is clear audio data with noise removed and the important visual frames. This preprocessing step prepares the data for analysis.

[0125] Step 3:

[0126] The server inputs pre-processed data into a deep learning model to perform sentiment and intent analysis. Using the audio data, a natural language processing model generates text data. Pre-processed audio and text are provided as input, and the analysis results are obtained as output. This makes it possible to evaluate the customer's emotional state and conversational intent.

[0127] Step 4:

[0128] The server integrates the analysis results and generates an optimal customer service strategy by utilizing information obtained from multiple data sources. The input consists of results from voice analysis, facial recognition, and gesture analysis, which are then integrated and outputted using a generated AI model to determine the best course of action. This results in a unified and effective strategy based on all the data.

[0129] Step 5:

[0130] The generated customer service strategy is visually fed back to smart glasses or displays connected to the terminal. The server takes detailed information about the strategy as input and provides the user with visualized advice to facilitate operation. This allows the user to quickly understand and implement specific customer service policies.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] The multimodal communication analysis system according to the present invention analyzes user emotions using various nonverbal information and provides an effective communication strategy. The embodiments for carrying out this invention are described below.

[0133] First, the user powers on the device and prepares it to use the system. The device is connected to a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting bodily movement information. The user uses these devices to collect the necessary data.

[0134] The device collects the user's voice as acoustic information in real time and captures facial expressions as visual information via a camera. It also monitors the user's movements using sensors to obtain information about their physical actions. Furthermore, the user provides the content of conversations and negotiations by inputting text information into the device.

[0135] The server analyzes the received audio, visual, bodily movement, and text information. First, it performs preprocessing to remove noise from the audio information and divide the visual information into frames. Next, it applies an emotion analysis engine to the audio and visual information. For the audio information, it recognizes the user's emotions by analyzing the tone, tempo, and volume of the voice. Similarly, it analyzes facial expressions from the visual information to determine the user's emotional state.

[0136] After the analysis is complete, the server integrates this information to evaluate the user's overall emotional state and negotiation situation. Based on this integrated data, the server generates optimal advice. The generated advice is presented to the user in an intuitively understandable format. Specifically, it is presented through graphs, text, and voice assistance displayed on the device screen.

[0137] For example, in a business negotiation, if the server detects that the other party is experiencing stress, it may suggest a break to ease the tension or offer advice on adjusting the content of the conversation. This allows the user to implement effective communication strategies in real time.

[0138] Furthermore, this invention anonymizes received information to strictly protect the privacy of user data. This allows users to use the system with peace of mind in a privacy-protected environment.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] The user powers on the device and configures it to start the multimodal communication analysis system. The device verifies that the microphone, camera, and sensors are functioning correctly and prepares to collect data.

[0142] Step 2:

[0143] The device uses a microphone to record the user's speech as acoustic information and a camera to record the user's facial expressions as visual information. It monitors the user's body movements through sensors and captures actual gestures and postures. In addition, it collects text information when the user inputs text.

[0144] Step 3:

[0145] The server receives raw data transmitted from the terminal in real time. The received data includes acoustic information, visual information, bodily movement information, and textual information.

[0146] Step 4:

[0147] The server preprocesses the received data. Acoustic information undergoes noise filtering, and visual information is divided into frames for image analysis. Furthermore, text data is processed using natural language processing techniques to convert it into a format that is easy to analyze.

[0148] Step 5:

[0149] The server inputs pre-processed data into the emotion analysis engine. It analyzes acoustic information to identify emotions from voice tone and tempo, and uses facial recognition technology to estimate emotional states from facial expressions.

[0150] Step 6:

[0151] The server integrates the analysis results. It combines the results from acoustic, visual, bodily movement, and textual information to assess the user's overall emotional state and understand the context of negotiations and conversations.

[0152] Step 7:

[0153] The server generates advice based on integrated analysis results. It uses deep learning models as needed to propose situation-sensitive communication strategies in real time.

[0154] Step 8:

[0155] The server sends the generated advice to the terminal. The terminal presents the advice to the user using visual and audio elements. Visual information includes graphs and text displayed on the dashboard, with audio guides supplementing necessary explanations.

[0156] Step 9:

[0157] Users use the provided advice and analysis results to choose their next action in actual negotiations and conversations. This allows users to achieve better communication outcomes.

[0158] (Example 2)

[0159] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0160] Conventional communication analysis systems have struggled to effectively integrate information from different modalities and accurately grasp users' emotional states. Furthermore, privacy concerns exist, necessitating secure data handling. It is essential to address these issues and provide a more effective and secure communication support system.

[0161] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0162] In this invention, the server includes data acquisition means for acquiring acoustic data, visual data, motion data, and text data; data preprocessing means for preprocessing the various data acquired by the data acquisition means; and data analysis means for analyzing the data preprocessed by the data preprocessing means. This enables the integrated analysis of information from different modalities, allowing for accurate understanding of the user's emotional state and secure information provision that takes privacy into consideration.

[0163] "Acoustic data" refers to information composed of the user's voice and other sounds, and serves as the basis for analyzing emotions and states through the characteristics of sound.

[0164] "Visual data" refers to information obtained through the user's facial expressions and movements, and is used to visually capture their emotional state and reactions.

[0165] "Motion data" refers to information obtained from the user's body movements and posture, and serves as material for judging nonverbal emotional expressions and intentions.

[0166] "Text data" refers to text information entered by users and is used to clarify the content and intent of conversations.

[0167] "Data acquisition means" refers to functions and devices for collecting acoustic, visual, motion, and textual data, and forms the foundation for acquiring system input information.

[0168] "Data preprocessing means" refers to the processing performed to convert acquired raw data into a format that is easy to analyze, and includes processes such as noise reduction and frame splitting.

[0169] "Data analysis means" refers to a function for analyzing the user's emotional state and reactions based on pre-processed data, and it evaluates information from each modality to derive results.

[0170] "Data integration means" refers to a function that integrates analyzed information to perform a comprehensive evaluation, and is used to determine the user's overall emotional state.

[0171] The "proposal generation method" refers to a function that creates advice and suggestions for users based on integrated analysis results, and is intended to support communication strategies.

[0172] "Information presentation means" refers to functions that visually or audibly convey generated suggestions or advice to the user, providing information in a way that is easy for the user to understand.

[0173] The system according to the present invention is designed to analyze multifaceted user data and provide effective communication strategies. This system acquires and analyzes acoustic, visual, motor, and written information to understand the user's emotional state.

[0174] The user starts using the system by activating the terminal. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting motion information. These hardware devices allow the terminal to acquire the necessary data in real time. Acoustic data is collected through the user's voice, visual data is captured using the camera to capture facial expressions, and motion data is collected from body movements by sensors. The user also inputs text information into the terminal to provide conversation content and other information.

[0175] The server first preprocesses the information received from the terminal to improve data quality. Specifically, it removes noise from audio data and divides visual data into frames. An emotion analysis engine is then applied to this preprocessed data, which analyzes the tone and tempo of speech, facial expressions, etc., to identify the user's emotional state and response.

[0176] Based on the integrated analysis results, the server uses a generative AI model to create optimal suggestions. These suggestions are sent to the terminal and provided to the user through information presentation tools. Users can receive advice in an intuitively understandable format through graphs, text, and voice assistance displayed on the screen.

[0177] For example, if the system analyzes that the other party is experiencing stress during a business negotiation, the server will generate advice suggesting an appropriate break for the user. A possible prompt might be, "What is a recommended break time when the other party is experiencing stress during a business negotiation?"

[0178] This invention further enhances the protection of personal information by performing de-identification processing to safeguard the confidentiality of user data. This mechanism provides users with an environment in which they can use the system with peace of mind.

[0179] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0180] Step 1:

[0181] The user activates the device and prepares to begin collecting acoustic, visual, motion, and text data. The device uses its microphone, camera, and sensors to acquire acoustic, visual, and motion data as input, while simultaneously collecting text data entered by the user. This prepares raw data on various aspects of communication.

[0182] Step 2:

[0183] The device removes noise from the acquired acoustic data and outputs clear audio data that preserves the tone and tempo of the voice. For visual data, the input from the camera is divided frame by frame and processed so that facial expressions can be clearly identified. For motion data, the raw data obtained from the sensors is organized based on the time series of motion to prepare accurate motion information. This preprocessing yields clean, structured data necessary for subsequent analysis steps.

[0184] Step 3:

[0185] The server receives pre-processed data and applies an emotion analysis engine. For acoustic data, it analyzes voice tone, tempo, and volume; for visual data, it analyzes facial expressions; and from motion data, it extracts nonverbal emotional indicators of the user. These analysis results are output individually to provide detailed insights into the user's emotional state.

[0186] Step 4:

[0187] The server integrates the analysis results and uses a generative AI model to comprehensively evaluate the information from each modality. This allows it to understand the user's overall emotional state and reactions, and prepare advice based on that information. The generated advice is then adjusted to the user's specific situation, enabling the implementation of the optimal communication strategy.

[0188] Step 5:

[0189] The server sends the final advice as data to the terminal and provides it to the user through information presentation tools. The terminal outputs the advice using graphs, text, and voice assistance, presenting the information in a way that the user can understand and act upon in real time. In addition, the raw data entered is anonymized during processing, contributing to the protection of user privacy.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] In modern society, security is increasingly important, but conventional security systems struggle to accurately grasp human emotional states and therefore fail to adequately detect potential risks. In this context, there is a need for technologies that can provide effective communication strategies in real time and improve security.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes a device for receiving acoustic information, visual information, and operational information; a processing unit for pre-processing the information acquired by the device; and an analysis unit for analyzing the information processed by the processing unit and determining the emotional state. This makes it possible to detect abnormalities early, generate and provide feedback based on security evaluations.

[0195] "Acoustic information" refers to audio data obtained from a subject and the results of its analysis.

[0196] "Visual information" refers to video data of the subject and the results of its analysis.

[0197] "Motion information" refers to data and analysis results related to the movements and posture of the subject's body.

[0198] "Receiving device" refers to a hardware device or equipment with related functions for acquiring acoustic information, visual information, and operational information.

[0199] A "pre-processing unit" refers to a computer system or group of programs that removes noise from received information and converts it into a format suitable for analysis.

[0200] The "analysis unit" is the part that performs algorithms and calculations to identify specific emotional states or patterns from pre-processed data.

[0201] A "security evaluation device" is a device or system that determines risks and detects anomalies based on analyzed information.

[0202] A "generation mechanism" refers to a means or process for generating user feedback based on information obtained from security evaluation equipment.

[0203] An "output device" is hardware or an interface that provides the generated feedback to the user.

[0204] To implement this invention, it is necessary to construct a system equipped with devices for receiving acoustic information, visual information, and motion information. First, for acoustic information, a high-sensitivity microphone is required to capture the target's voice in real time. Next, for visual information, a high-resolution camera is used to capture the target's face and movements. Furthermore, to acquire motion information, a motion sensor is incorporated to analyze the target's body movements.

[0205] Upon receiving this data, the server first uses a preprocessor to remove noise and normalize the data, and then the analysis unit determines the emotional state. The analysis unit is equipped with speech processing software (e.g., Google® Speech-to-Text) and image recognition software (e.g., Microsoft® Azure® Face API). This allows the server to determine emotions from the tone and tempo of acoustic information, changes in facial expressions from visual information, and body movements.

[0206] Based on these results, the security assessment device detects anomalies. For example, in airport security, if a visitor shows signs of potential stress or anxiety, the security assessment device recognizes this and generates corresponding feedback. The generation mechanism constructs appropriate advice or warnings based on the detected information and presents them to the user through the output device.

[0207] The following is a concrete example of a prompt statement generated using a generative AI model:

[0208] "Using multimodal data, we assess visitors' emotional states and determine if they are experiencing increased anxiety or stress. If an anomaly is detected, security personnel are notified in real time."

[0209] This system makes it possible to detect and respond to risks more effectively than conventional security systems.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The user activates the device and prepares various sensors to collect acoustic, visual, and motion information. At this stage, the user's voice is captured by the microphone, and facial images are collected by the camera. Motion sensors also record the user's movements in real time. This data is then transmitted from the device to the server.

[0213] Step 2:

[0214] The server preprocesses the received data. Specifically, it denoises the audio data and breaks down the visual data into frames. Here, the input is raw audio, visual, and motion information, and the output is data that has been formatted to be analyzable.

[0215] Step 3:

[0216] The server analyzes the pre-processed data using speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API) to identify the user's emotional state. The input is pre-processed acoustic and visual information, and the output is an indicator of the user's emotional state. In this step, emotions are analyzed based on factors such as voice tone and facial expression changes.

[0217] Step 4:

[0218] The server uses a security evaluation device to detect anomalies based on the analysis results. The input here is the result obtained from emotion analysis, and the output is an evaluation value indicating the degree of risk. This process generates a warning, especially if high levels of stress or anxiety are detected.

[0219] Step 5:

[0220] The server generates anomaly-based feedback using a generation mechanism. The input is an anomaly detection evaluation value, and the output is specific advice or warning messages presented to the user. The feedback is evaluated in real time and shaped appropriately.

[0221] Step 6:

[0222] The device provides feedback to the user via an output device. This feedback is presented through the device's screen or audio output. For example, a warning message such as, "Your stress level is high. Additional checks may be needed," might be displayed.

[0223] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0224] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0225] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0226] [Second Embodiment]

[0227] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0228] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0229] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0230] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0231] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0233] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0234] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0235] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0236] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0237] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0238] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0239] The multimodal communication analysis AI agent according to the present invention is a system that enables a comprehensive understanding of nonverbal elements in negotiation and dialogue situations by receiving, analyzing, integrating various types of information and providing advice on optimal strategies. Its embodiments are described below.

[0240] First, the user activates the system and collects the necessary data through the terminal. The terminal picks up participants' voices as acoustic information using a microphone and captures video as visual information using a camera. It also obtains physical movement information by detecting the user's movements with sensors. Finally, it obtains text information by inputting the content of negotiations and conversations as text.

[0241] The server receives this information in real time and performs effective preprocessing. Specifically, it removes noise from acoustic data and extracts necessary frames from visual data. It also performs text analysis on text data using natural language processing techniques. Body movement information is converted into a format that can be analyzed by motion recognition algorithms.

[0242] During the analysis phase, the server analyzes each piece of pre-processed information based on its specific characteristics. Voice analysis evaluates participants' emotions and intentions, while facial expression analysis captures subtle facial changes. Gesture analysis tracks hand and body movements and extracts important ones. Text analysis understands the tone and intent of speech to grasp the overall picture.

[0243] Next, the server comprehensively evaluates these analysis results to capture the participants' emotional states and the atmosphere of the negotiation in a unified manner. Based on this integrated analysis, the server generates appropriate advice. The advice generation utilizes deep learning techniques and past pattern analysis results to propose the optimal strategy for the specific situation the user is facing.

[0244] The generated advice is provided to the user through the device. The user receives visual feedback on the device screen and can refer to audio guides. This allows the user to immediately obtain useful information and effectively advance negotiations and discussions.

[0245] Furthermore, the server ensures that all data is anonymized and securely managed to protect user privacy. This allows users to use the system with peace of mind.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The user activates the terminal and starts the multimodal communication analysis system. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting body movement information.

[0249] Step 2:

[0250] The device begins data collection upon receiving a start command from the user. It collects acoustic information using a microphone, acquires visual information using a camera, and records body movement information using sensors. For text information, the user inputs the text into the device.

[0251] Step 3:

[0252] The server receives raw data transmitted from the terminal. This received data includes acoustic, visual, bodily motion, and textual information.

[0253] Step 4:

[0254] The server preprocesses the received data. Noise reduction is performed on acoustic data, and frame division is carried out on visual data. Body movement data is formatted into a unified sensor data format, and text data is converted into a form suitable for language analysis using natural language processing techniques.

[0255] Step 5:

[0256] The server analyzes the pre-processed data. It performs emotion analysis on acoustic information and facial expression analysis on visual information. It performs gesture analysis on bodily movement information and analyzes intent and context on written information.

[0257] Step 6:

[0258] The server integrates the results of each analysis. Based on the integrated information, it evaluates the overall communication situation and understands the participants' emotions and circumstances.

[0259] Step 7:

[0260] The server generates advice from integrated data. It uses deep learning and analysis of historical patterns to suggest optimal negotiation strategies and actions for the user.

[0261] Step 8:

[0262] The server sends the generated advice to the terminal. The terminal presents the results, including the advice, to the user in visual and auditory formats.

[0263] Step 9:

[0264] Users receive advice and analysis results displayed on their devices and decide on their next actions based on them. This enables users to communicate effectively in real time.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] Nonverbal elements play a crucial role in modern negotiations and dialogues. However, comprehensively understanding and effectively analyzing this diverse information is challenging. In particular, there is a need to accurately grasp participants' emotional states and the atmosphere of negotiations in real time and provide appropriate advice based on that understanding, but existing technologies are unable to meet this challenge.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes acquisition means for acquiring audio data, video data, body movement data, and text data; a data processing mechanism for preprocessing the data obtained by the acquisition means; and data analysis means for analyzing the preprocessed data provided by the data processing mechanism. This enables centralized analysis of diverse information and the generation of optimal advice.

[0270] "Acquisition means" refers to a mechanism for collecting audio data, video data, body movement data, and text data and providing them to a server.

[0271] A "data processing mechanism" is a device or method that organizes data obtained through acquisition means and prepares it in a format that can be analyzed.

[0272] A "data analysis tool" is a system or process for analyzing pre-processed data and extracting and evaluating specific information based on its content.

[0273] An "analysis integration tool" is a function that integrates information obtained from individual data analyses and evaluates it from a more holistic perspective.

[0274] A "content generation device" is a mechanism that creates suggestions and advice for users based on integrated analysis results.

[0275] An "information transmission system" is a means of conveying the generated proposal content to users visually or audibly.

[0276] The "evaluation function" is a function within the analysis and integration system that determines the characteristics of emotional states and reactions based on the obtained data.

[0277] "Anonymization" is a technique or method that removes or conceals personally identifiable information from data in order to protect privacy.

[0278] This multimodal communication analysis system is designed to enable users to effectively acquire and analyze information in negotiations and dialogues, and to obtain optimal advice. The system has the functionality to collect and analyze audio data, video data, bodily motion data, and text data.

[0279] The user activates the system and collects necessary data using a terminal. The terminal has a built-in microphone, camera, and motion sensor, and uses this hardware to acquire voice, video, and body movement information. In addition, the content of negotiations and conversations can be entered as text data via a keyboard or other input devices.

[0280] The server receives data from terminals in real time and performs preprocessing using a data processing mechanism. Preprocessing includes noise reduction for audio data, frame extraction for video data, and natural language processing for text data. This prepares the data for analysis.

[0281] The analyzed data is integrated and analyzed using server-generated AI models to understand the emotional state of participants and the atmosphere of negotiations. Based on these analysis results, optimal advice is generated. The advice is provided as the most appropriate response for the current situation, based on a model that has learned past data patterns.

[0282] The device provides feedback to the user with the generated advice. Visual feedback is displayed on the screen, and audio guidance is used as needed. For example, it can show specific negotiation strategies or provide the results of an analysis of the user's emotional state.

[0283] Regarding privacy protection, the server anonymizes all data and securely manages user information. With this protection function, users can use the system with confidence.

[0284] As a specific example, a situation such as "When the client expresses concern about the price of a new product, I hope to be advised on how to emphasize the added value of the product instead of lowering the price" can be considered. An example of the prompt text in this case is as follows:

[0285] Ask the AI agent: When the client expresses concern about the price, how can I emphasize the added value of the product without lowering it?

[0286] By such a method, this system provides immediate and effective support for the issues faced by users.

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The user activates the terminal and collects the data necessary for negotiation or interaction. The input includes acoustic data obtained by the microphone of the terminal, video data captured by the camera, body movement data detected by the sensor, and text data input by the user. The output is a composite data stream transmitted to the server.

[0290] Step 2:

[0291] The server obtains the composite data stream received from the terminal and performs preprocessing with the data processing mechanism. Specific operations include noise removal for the input of acoustic data, extraction of necessary frames from video data, and language analysis by natural language processing for text data. The output of this processing is a data set in a form suitable for analysis.

[0292] Step 3:

[0293] The server analyzes the pre-processed data using data analysis tools. Based on each data input, it analyzes emotional nuances from audio data, changes in facial expressions and gestures from video data, and intentions and tone from text data. The output of this step is the analysis results, such as emotional states.

[0294] Step 4:

[0295] The server integrates the analysis results obtained from the data analysis means using the analysis integration means. The analysis results are unified, summarizing the overall atmosphere of the negotiation and the emotional state of the participants. The output of this integration process is the integrated evaluation result.

[0296] Step 5:

[0297] The server generates optimal advice using a generative AI model based on the integrated evaluation results. Based on the integrated results, it devises countermeasures for specific situations the user faces and creates advice statements. The output is the specific advice provided to the user.

[0298] Step 6:

[0299] The terminal provides the user with advice from the server. The advice is displayed visually on the screen and accompanied by specific feedback actions explained by voice guidance. The final output of this step is easy-to-understand advice received by the user.

[0300] Step 7:

[0301] The server anonymizes the acquired data, ensuring secure information management while protecting privacy. Each data element is anonymized and kept inaccessible from the outside. The output of this process is an anonymized, secure data record.

[0302] (Application Example 1)

[0303] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0304] In many physical stores, it is difficult for salespersons to accurately understand the emotions and intentions of customers, and as a result, appropriate customer service may not be provided. In addition, since there is a lack of tools for salespersons to analyze customer reactions in real time and propose optimal customer service methods, there are problems in improving customer satisfaction and maximizing sales opportunities.

[0305] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0306] In this invention, the server includes means for receiving acoustic data, visual data, body motion data, and text data, means for preprocessing various data acquired by the receiving means, and means for analyzing the data preprocessed by the preprocessing means. Thereby, it becomes possible to analyze the emotional state and reaction data in real time during the interaction with the customer and propose an optimal customer service method.

[0307] "Acoustic data" is voice information acquired via a voice input device such as a microphone, and is data including sound information related to a customer's speech or conversation.

[0308] "Visual data" is video information acquired by a video input device such as a camera, and is data referring to a series of visual information including a customer's facial expression and movements.

[0309] "Body motion data" is data including information on the movements and postures of a person's body obtained by an acceleration sensor or a motion sensor.

[0310] "Text data" is data including information obtained by converting voice into character information by voice recognition technology and characterizing a customer's speech.

[0311] "Analysis results" refer to information obtained after analyzing acoustic data, visual data, bodily movement data, and text data, including evaluations of the customer's emotional state and intentions.

[0312] A "strategy generation tool" is a device or algorithm that performs a series of processes to generate an optimal communication strategy based on analysis results.

[0313] A "presentation means" is a means of showing a generated communication strategy to the user, and is a device that transmits information visually or audibly using displays and audio equipment.

[0314] "Optimization suggestion means" refers to a system or process that makes specific suggestions for improving customer service methods based on real-time customer feedback.

[0315] This shows an embodiment for carrying out the invention.

[0316] To realize this application example, the present invention is configured as follows: The server provides an advanced analysis system to support the interaction between the customer and the salesperson. First, acoustic data, visual data, bodily motion data, and text data are acquired using a receiving device. Acoustic data is obtained by recording the customer's voice using a microphone and converting it into text data using speech recognition technology. Visual data is obtained by capturing the customer's facial expressions and gestures with a camera and analyzing it using image recognition technology. Bodily motion data is obtained by detecting the customer's movements with a motion sensor and processing it with a motion analysis algorithm.

[0317] The server processes this data in real time and performs detailed analysis based on the characteristics of each data point using deep learning models. By integrating these analysis results and evaluating customer emotions and intentions, the most effective customer service strategy is generated. The generated strategy is presented to the sales staff and visually fed back using smart glasses or display devices.

[0318] Furthermore, for particularly complex customer requests, adaptive learning of deep learning models can be utilized to improve analysis accuracy. This system also incorporates data anonymization, providing strong protection for customer privacy.

[0319] For example, if analysis of customer voices detects that the customer is excited, the system will advise the salesperson to "explain things carefully to the customer and take your time guiding them." As an example of a prompt, it will generate a question such as, "The customer's tone of voice has started to change. What emotions could this indicate?" to draw the salesperson's attention.

[0320] Thus, the present invention makes it possible to increase customer satisfaction and expand sales opportunities.

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The user uses the device's camera and microphone to acquire customer audio data, visual data, body movement data, and conversation data. Audio and visual information are captured as input. This collects basic data on customer interactions.

[0324] Step 2:

[0325] To analyze the received data, the server first removes noise from the audio data and extracts the necessary frames from the visual data. The input here is the original audio and video, and the output is clear audio data with noise removed and the important visual frames. This preprocessing step prepares the data for analysis.

[0326] Step 3:

[0327] The server inputs pre-processed data into a deep learning model to perform sentiment and intent analysis. Using the audio data, a natural language processing model generates text data. Pre-processed audio and text are provided as input, and the analysis results are obtained as output. This makes it possible to evaluate the customer's emotional state and conversational intent.

[0328] Step 4:

[0329] The server integrates the analysis results and generates an optimal customer service strategy by utilizing information obtained from multiple data sources. The input consists of results from voice analysis, facial recognition, and gesture analysis, which are then integrated and outputted using a generated AI model to determine the best course of action. This results in a unified and effective strategy based on all the data.

[0330] Step 5:

[0331] The generated customer service strategy is visually fed back to smart glasses or displays connected to the terminal. The server takes detailed information about the strategy as input and provides the user with visualized advice to facilitate operation. This allows the user to quickly understand and implement specific customer service policies.

[0332] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0333] The multimodal communication analysis system according to the present invention analyzes user emotions using various nonverbal information and provides an effective communication strategy. The embodiments for carrying out this invention are described below.

[0334] First, the user powers on the device and prepares it to use the system. The device is connected to a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting bodily movement information. The user uses these devices to collect the necessary data.

[0335] The device collects the user's voice as acoustic information in real time and captures facial expressions as visual information via a camera. It also monitors the user's movements using sensors to obtain information about their physical actions. Furthermore, the user provides the content of conversations and negotiations by inputting text information into the device.

[0336] The server analyzes the received audio, visual, bodily movement, and text information. First, it performs preprocessing to remove noise from the audio information and divide the visual information into frames. Next, it applies an emotion analysis engine to the audio and visual information. For the audio information, it recognizes the user's emotions by analyzing the tone, tempo, and volume of the voice. Similarly, it analyzes facial expressions from the visual information to determine the user's emotional state.

[0337] After the analysis is complete, the server integrates this information to evaluate the user's overall emotional state and negotiation situation. Based on this integrated data, the server generates optimal advice. The generated advice is presented to the user in an intuitively understandable format. Specifically, it is presented through graphs, text, and voice assistance displayed on the device screen.

[0338] For example, in a business negotiation, if the server detects that the other party is experiencing stress, it may suggest a break to ease the tension or offer advice on adjusting the content of the conversation. This allows the user to implement effective communication strategies in real time.

[0339] Furthermore, this invention anonymizes received information to strictly protect the privacy of user data. This allows users to use the system with peace of mind in a privacy-protected environment.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The user powers on the device and configures it to start the multimodal communication analysis system. The device verifies that the microphone, camera, and sensors are functioning correctly and prepares to collect data.

[0343] Step 2:

[0344] The device uses a microphone to record the user's speech as acoustic information and a camera to record the user's facial expressions as visual information. It monitors the user's body movements through sensors and captures actual gestures and postures. In addition, it collects text information when the user inputs text.

[0345] Step 3:

[0346] The server receives raw data transmitted from the terminal in real time. The received data includes acoustic information, visual information, bodily movement information, and textual information.

[0347] Step 4:

[0348] The server preprocesses the received data. Acoustic information undergoes noise filtering, and visual information is divided into frames for image analysis. Furthermore, text data is processed using natural language processing techniques to convert it into a format that is easy to analyze.

[0349] Step 5:

[0350] The server inputs pre-processed data into the emotion analysis engine. It analyzes acoustic information to identify emotions from voice tone and tempo, and uses facial recognition technology to estimate emotional states from facial expressions.

[0351] Step 6:

[0352] The server integrates the analysis results. It combines the results from acoustic, visual, bodily movement, and textual information to assess the user's overall emotional state and understand the context of negotiations and conversations.

[0353] Step 7:

[0354] The server generates advice based on integrated analysis results. It uses deep learning models as needed to propose situation-sensitive communication strategies in real time.

[0355] Step 8:

[0356] The server sends the generated advice to the terminal. The terminal presents the advice to the user using visual and audio elements. Visual information includes graphs and text displayed on the dashboard, with audio guides supplementing necessary explanations.

[0357] Step 9:

[0358] Users use the provided advice and analysis results to choose their next action in actual negotiations and conversations. This allows users to achieve better communication outcomes.

[0359] (Example 2)

[0360] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0361] Conventional communication analysis systems have struggled to effectively integrate information from different modalities and accurately grasp users' emotional states. Furthermore, privacy concerns exist, necessitating secure data handling. It is essential to address these issues and provide a more effective and secure communication support system.

[0362] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0363] In this invention, the server includes data acquisition means for acquiring acoustic data, visual data, motion data, and text data; data preprocessing means for preprocessing the various data acquired by the data acquisition means; and data analysis means for analyzing the data preprocessed by the data preprocessing means. This enables the integrated analysis of information from different modalities, allowing for accurate understanding of the user's emotional state and secure information provision that takes privacy into consideration.

[0364] "Acoustic data" refers to information composed of the user's voice and other sounds, and serves as the basis for analyzing emotions and states through the characteristics of sound.

[0365] "Visual data" refers to information obtained through the user's facial expressions and movements, and is used to visually capture their emotional state and reactions.

[0366] "Motion data" refers to information obtained from the user's body movements and posture, and serves as material for judging nonverbal emotional expressions and intentions.

[0367] "Text data" refers to text information entered by users and is used to clarify the content and intent of conversations.

[0368] "Data acquisition means" refers to functions and devices for collecting acoustic, visual, motion, and textual data, and forms the foundation for acquiring system input information.

[0369] "Data preprocessing means" refers to the processing performed to convert acquired raw data into a format that is easy to analyze, and includes processes such as noise reduction and frame splitting.

[0370] "Data analysis means" refers to a function for analyzing the user's emotional state and reactions based on pre-processed data, and it evaluates information from each modality to derive results.

[0371] "Data integration means" refers to a function that integrates analyzed information to perform a comprehensive evaluation, and is used to determine the user's overall emotional state.

[0372] The "proposal generation method" refers to a function that creates advice and suggestions for users based on integrated analysis results, and is intended to support communication strategies.

[0373] "Information presentation means" refers to functions that visually or audibly convey generated suggestions or advice to the user, providing information in a way that is easy for the user to understand.

[0374] The system according to the present invention is designed to analyze multifaceted user data and provide effective communication strategies. This system acquires and analyzes acoustic, visual, motor, and written information to understand the user's emotional state.

[0375] The user starts using the system by activating the terminal. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting motion information. These hardware devices allow the terminal to acquire the necessary data in real time. Acoustic data is collected through the user's voice, visual data is captured using the camera to capture facial expressions, and motion data is collected from body movements by sensors. The user also inputs text information into the terminal to provide conversation content and other information.

[0376] The server first preprocesses the information received from the terminal to improve data quality. Specifically, it removes noise from audio data and divides visual data into frames. An emotion analysis engine is then applied to this preprocessed data, which analyzes the tone and tempo of speech, facial expressions, etc., to identify the user's emotional state and response.

[0377] Based on the integrated analysis results, the server uses a generative AI model to create optimal suggestions. These suggestions are sent to the terminal and provided to the user through information presentation tools. Users can receive advice in an intuitively understandable format through graphs, text, and voice assistance displayed on the screen.

[0378] For example, if the system analyzes that the other party is experiencing stress during a business negotiation, the server will generate advice suggesting an appropriate break for the user. A possible prompt might be, "What is a recommended break time when the other party is experiencing stress during a business negotiation?"

[0379] This invention further enhances the protection of personal information by performing de-identification processing to safeguard the confidentiality of user data. This mechanism provides users with an environment in which they can use the system with peace of mind.

[0380] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0381] Step 1:

[0382] The user activates the device and prepares to begin collecting acoustic, visual, motion, and text data. The device uses its microphone, camera, and sensors to acquire acoustic, visual, and motion data as input, while simultaneously collecting text data entered by the user. This prepares raw data on various aspects of communication.

[0383] Step 2:

[0384] The device removes noise from the acquired acoustic data and outputs clear audio data that preserves the tone and tempo of the voice. For visual data, the input from the camera is divided frame by frame and processed so that facial expressions can be clearly identified. For motion data, the raw data obtained from the sensors is organized based on the time series of motion to prepare accurate motion information. This preprocessing yields clean, structured data necessary for subsequent analysis steps.

[0385] Step 3:

[0386] The server receives pre-processed data and applies an emotion analysis engine. For acoustic data, it analyzes voice tone, tempo, and volume; for visual data, it analyzes facial expressions; and from motion data, it extracts nonverbal emotional indicators of the user. These analysis results are output individually to provide detailed insights into the user's emotional state.

[0387] Step 4:

[0388] The server integrates the analysis results and uses a generative AI model to comprehensively evaluate the information from each modality. This allows it to understand the user's overall emotional state and reactions, and prepare advice based on that information. The generated advice is then adjusted to the user's specific situation, enabling the implementation of the optimal communication strategy.

[0389] Step 5:

[0390] The server sends the final advice as data to the terminal and provides it to the user through information presentation tools. The terminal outputs the advice using graphs, text, and voice assistance, presenting the information in a way that the user can understand and act upon in real time. In addition, the raw data entered is anonymized during processing, contributing to the protection of user privacy.

[0391] (Application Example 2)

[0392] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0393] In modern society, security is increasingly important, but conventional security systems struggle to accurately grasp human emotional states and therefore fail to adequately detect potential risks. In this context, there is a need for technologies that can provide effective communication strategies in real time and improve security.

[0394] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0395] In this invention, the server includes a device for receiving acoustic information, visual information, and operational information; a processing unit for pre-processing the information acquired by the device; and an analysis unit for analyzing the information processed by the processing unit and determining the emotional state. This makes it possible to detect abnormalities early, generate and provide feedback based on security evaluations.

[0396] "Acoustic information" refers to audio data obtained from a subject and the results of its analysis.

[0397] "Visual information" refers to video data of the subject and the results of its analysis.

[0398] "Motion information" refers to data and analysis results related to the movements and posture of the subject's body.

[0399] "Receiving device" refers to a hardware device or equipment with related functions for acquiring acoustic information, visual information, and operational information.

[0400] A "pre-processing unit" refers to a computer system or group of programs that removes noise from received information and converts it into a format suitable for analysis.

[0401] The "analysis unit" is the part that performs algorithms and calculations to identify specific emotional states or patterns from pre-processed data.

[0402] A "security evaluation device" is a device or system that determines risks and detects anomalies based on analyzed information.

[0403] A "generation mechanism" refers to a means or process for generating user feedback based on information obtained from security evaluation equipment.

[0404] An "output device" is hardware or an interface that provides the generated feedback to the user.

[0405] To implement this invention, it is necessary to construct a system equipped with devices for receiving acoustic information, visual information, and motion information. First, for acoustic information, a high-sensitivity microphone is required to capture the target's voice in real time. Next, for visual information, a high-resolution camera is used to capture the target's face and movements. Furthermore, to acquire motion information, a motion sensor is incorporated to analyze the target's body movements.

[0406] Upon receiving this data, the server first uses a preprocessor to remove noise and normalize the data, and then the analysis unit determines the emotional state. The analysis unit is equipped with speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API). This allows the server to determine emotions from the tone and tempo of the acoustic information, changes in facial expressions from visual information, and body movements.

[0407] Based on these results, the security assessment device detects anomalies. For example, in airport security, if a visitor shows signs of potential stress or anxiety, the security assessment device recognizes this and generates corresponding feedback. The generation mechanism constructs appropriate advice or warnings based on the detected information and presents them to the user through the output device.

[0408] The following is a concrete example of a prompt statement generated using a generative AI model:

[0409] "Using multimodal data, we assess visitors' emotional states and determine if they are experiencing increased anxiety or stress. If an anomaly is detected, security personnel are notified in real time."

[0410] This system makes it possible to detect and respond to risks more effectively than conventional security systems.

[0411] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0412] Step 1:

[0413] The user activates the device and prepares various sensors to collect acoustic, visual, and motion information. At this stage, the user's voice is captured by the microphone, and facial images are collected by the camera. Motion sensors also record the user's movements in real time. This data is then transmitted from the device to the server.

[0414] Step 2:

[0415] The server preprocesses the received data. Specifically, it denoises the audio data and breaks down the visual data into frames. Here, the input is raw audio, visual, and motion information, and the output is data that has been formatted to be analyzable.

[0416] Step 3:

[0417] The server analyzes the pre-processed data using speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API) to identify the user's emotional state. The input is pre-processed acoustic and visual information, and the output is an indicator of the user's emotional state. In this step, emotions are analyzed based on factors such as voice tone and facial expression changes.

[0418] Step 4:

[0419] The server uses a security evaluation device to detect anomalies based on the analysis results. The input here is the result obtained from emotion analysis, and the output is an evaluation value indicating the degree of risk. This process generates a warning, especially if high levels of stress or anxiety are detected.

[0420] Step 5:

[0421] The server generates anomaly-based feedback using a generation mechanism. The input is an anomaly detection evaluation value, and the output is specific advice or warning messages presented to the user. The feedback is evaluated in real time and shaped appropriately.

[0422] Step 6:

[0423] The device provides feedback to the user via an output device. This feedback is presented through the device's screen or audio output. For example, a warning message such as, "Your stress level is high. Additional checks may be needed," might be displayed.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0425] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0427] [Third Embodiment]

[0428] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0429] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0430] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0432] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0434] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0435] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0436] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0439] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0440] The multimodal communication analysis AI agent according to the present invention is a system that enables a comprehensive understanding of nonverbal elements in negotiation and dialogue situations by receiving, analyzing, integrating various types of information and providing advice on optimal strategies. Its embodiments are described below.

[0441] First, the user activates the system and collects the necessary data through the terminal. The terminal picks up participants' voices as acoustic information using a microphone and captures video as visual information using a camera. It also obtains physical movement information by detecting the user's movements with sensors. Finally, it obtains text information by inputting the content of negotiations and conversations as text.

[0442] The server receives this information in real time and performs effective preprocessing. Specifically, it removes noise from acoustic data and extracts necessary frames from visual data. It also performs text analysis on text data using natural language processing techniques. Body movement information is converted into a format that can be analyzed by motion recognition algorithms.

[0443] During the analysis phase, the server analyzes each piece of pre-processed information based on its specific characteristics. Voice analysis evaluates participants' emotions and intentions, while facial expression analysis captures subtle facial changes. Gesture analysis tracks hand and body movements and extracts important ones. Text analysis understands the tone and intent of speech to grasp the overall picture.

[0444] Next, the server comprehensively evaluates these analysis results to capture the participants' emotional states and the atmosphere of the negotiation in a unified manner. Based on this integrated analysis, the server generates appropriate advice. The advice generation utilizes deep learning techniques and past pattern analysis results to propose the optimal strategy for the specific situation the user is facing.

[0445] The generated advice is provided to the user through the device. The user receives visual feedback on the device screen and can refer to audio guides. This allows the user to immediately obtain useful information and effectively advance negotiations and discussions.

[0446] Furthermore, the server ensures that all data is anonymized and securely managed to protect user privacy. This allows users to use the system with peace of mind.

[0447] The following describes the processing flow.

[0448] Step 1:

[0449] The user activates the terminal and starts the multimodal communication analysis system. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting body movement information.

[0450] Step 2:

[0451] The device begins data collection upon receiving a start command from the user. It collects acoustic information using a microphone, acquires visual information using a camera, and records body movement information using sensors. For text information, the user inputs the text into the device.

[0452] Step 3:

[0453] The server receives raw data transmitted from the terminal. This received data includes acoustic, visual, bodily motion, and textual information.

[0454] Step 4:

[0455] The server preprocesses the received data. Noise reduction is performed on acoustic data, and frame division is carried out on visual data. Body movement data is formatted into a unified sensor data format, and text data is converted into a form suitable for language analysis using natural language processing techniques.

[0456] Step 5:

[0457] The server analyzes the pre-processed data. It performs emotion analysis on acoustic information and facial expression analysis on visual information. It performs gesture analysis on bodily movement information and analyzes intent and context on written information.

[0458] Step 6:

[0459] The server integrates the results of each analysis. Based on the integrated information, it evaluates the overall communication situation and understands the participants' emotions and circumstances.

[0460] Step 7:

[0461] The server generates advice from integrated data. It uses deep learning and analysis of historical patterns to suggest optimal negotiation strategies and actions for the user.

[0462] Step 8:

[0463] The server sends the generated advice to the terminal. The terminal presents the results, including the advice, to the user in visual and auditory formats.

[0464] Step 9:

[0465] Users receive advice and analysis results displayed on their devices and decide on their next actions based on them. This enables users to communicate effectively in real time.

[0466] (Example 1)

[0467] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0468] Nonverbal elements play a crucial role in modern negotiations and dialogues. However, comprehensively understanding and effectively analyzing this diverse information is challenging. In particular, there is a need to accurately grasp participants' emotional states and the atmosphere of negotiations in real time and provide appropriate advice based on that understanding, but existing technologies are unable to meet this challenge.

[0469] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0470] In this invention, the server includes acquisition means for acquiring audio data, video data, body movement data, and text data; a data processing mechanism for preprocessing the data obtained by the acquisition means; and data analysis means for analyzing the preprocessed data provided by the data processing mechanism. This enables centralized analysis of diverse information and the generation of optimal advice.

[0471] "Acquisition means" refers to a mechanism for collecting audio data, video data, body movement data, and text data and providing them to a server.

[0472] A "data processing mechanism" is a device or method that organizes data obtained through acquisition means and prepares it in a format that can be analyzed.

[0473] A "data analysis tool" is a system or process for analyzing pre-processed data and extracting and evaluating specific information based on its content.

[0474] An "analysis integration tool" is a function that integrates information obtained from individual data analyses and evaluates it from a more holistic perspective.

[0475] A "content generation device" is a mechanism that creates suggestions and advice for users based on integrated analysis results.

[0476] An "information transmission system" is a means of conveying the generated proposal content to users visually or audibly.

[0477] The "evaluation function" is a function within the analysis and integration system that determines the characteristics of emotional states and reactions based on the obtained data.

[0478] "Anonymization" is a technique or method that removes or conceals personally identifiable information from data in order to protect privacy.

[0479] This multimodal communication analysis system is designed to enable users to effectively acquire and analyze information in negotiations and dialogues, and to obtain optimal advice. The system has the functionality to collect and analyze audio data, video data, bodily motion data, and text data.

[0480] The user activates the system and collects necessary data using a terminal. The terminal has a built-in microphone, camera, and motion sensor, and uses this hardware to acquire voice, video, and body movement information. In addition, the content of negotiations and conversations can be entered as text data via a keyboard or other input devices.

[0481] The server receives data from terminals in real time and performs preprocessing using a data processing mechanism. Preprocessing includes noise reduction for audio data, frame extraction for video data, and natural language processing for text data. This prepares the data for analysis.

[0482] The analyzed data is integrated and analyzed using server-generated AI models to understand the emotional state of participants and the atmosphere of negotiations. Based on these analysis results, optimal advice is generated. The advice is provided as the most appropriate response for the current situation, based on a model that has learned past data patterns.

[0483] The device provides feedback to the user with the generated advice. Visual feedback is displayed on the screen, and audio guidance is used as needed. For example, it can show specific negotiation strategies or provide the results of an analysis of the user's emotional state.

[0484] Regarding privacy protection, the server anonymizes all data and securely manages user information. This protection feature allows users to use the system with peace of mind.

[0485] A concrete example would be a situation where, "a client is concerned about the price of a new product, and we need advice on how to emphasize the product's added value instead of lowering the price." An example of a prompt in this case would be:

[0486] Question for the AI ​​agent: If a client expresses concerns about pricing, how can I emphasize the added value of the product without lowering the price?

[0487] In this way, the system provides immediate and effective support for the challenges that users face.

[0488] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0489] Step 1:

[0490] The user activates the device and collects the data necessary for negotiations and conversations. Inputs include acoustic data acquired by the device's microphone, video data captured by the camera, body movement data detected by sensors, and text data entered by the user. Output is a composite data stream sent to the server.

[0491] Step 2:

[0492] The server receives a composite data stream from the terminal and performs preprocessing using a data processing mechanism. Specific operations include noise reduction on audio data input, extraction of necessary frames from video data, and natural language processing for text data. The output of this processing is a dataset in a format suitable for analysis.

[0493] Step 3:

[0494] The server analyzes the pre-processed data using data analysis tools. Based on each data input, it analyzes emotional nuances from audio data, changes in facial expressions and gestures from video data, and intentions and tone from text data. The output of this step is the analysis results, such as emotional states.

[0495] Step 4:

[0496] The server integrates the analysis results obtained from the data analysis means using the analysis integration means. The analysis results are unified, summarizing the overall atmosphere of the negotiation and the emotional state of the participants. The output of this integration process is the integrated evaluation result.

[0497] Step 5:

[0498] The server generates optimal advice using a generative AI model based on the integrated evaluation results. Based on the integrated results, it devises countermeasures for specific situations the user faces and creates advice statements. The output is the specific advice provided to the user.

[0499] Step 6:

[0500] The terminal provides the user with advice from the server. The advice is displayed visually on the screen and accompanied by specific feedback actions explained by voice guidance. The final output of this step is easy-to-understand advice received by the user.

[0501] Step 7:

[0502] The server anonymizes the acquired data, ensuring secure information management while protecting privacy. Each data element is anonymized and kept inaccessible from the outside. The output of this process is an anonymized, secure data record.

[0503] (Application Example 1)

[0504] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0505] In many brick-and-mortar stores, sales staff often struggle to accurately understand customers' emotions and intentions, resulting in inadequate customer service. Furthermore, a lack of tools to enable sales staff to analyze customer reactions in real time and propose optimal service approaches poses challenges to improving customer satisfaction and maximizing sales opportunities.

[0506] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0507] In this invention, the server includes means for receiving acoustic data, visual data, bodily movement data, and text data; means for preprocessing the various data acquired by the receiving means; and means for analyzing the data preprocessed by the preprocessing means. This makes it possible to analyze emotional state and reaction data in real time during interaction with customers and propose the optimal customer service method.

[0508] "Audio data" refers to audio information acquired via audio input devices such as microphones, and includes audio information related to customer speech and conversations.

[0509] "Visual data" refers to video information acquired by video input devices such as cameras, and includes a series of visual information such as a customer's facial expressions and movements.

[0510] "Body movement data" refers to data that includes information about a person's body movements and posture obtained from acceleration sensors and motion sensors.

[0511] "Text data" refers to data that includes information obtained by converting speech into text information using speech recognition technology, and which includes information that is a transcription of customer utterances.

[0512] "Analysis results" refer to information obtained after analyzing acoustic data, visual data, bodily movement data, and text data, including evaluations of the customer's emotional state and intentions.

[0513] A "strategy generation tool" is a device or algorithm that performs a series of processes to generate an optimal communication strategy based on analysis results.

[0514] A "presentation means" is a means of showing a generated communication strategy to the user, and is a device that transmits information visually or audibly using displays and audio equipment.

[0515] "Optimization suggestion means" refers to a system or process that makes specific suggestions for improving customer service methods based on real-time customer feedback.

[0516] This shows an embodiment for carrying out the invention.

[0517] To realize this application example, the present invention is configured as follows: The server provides an advanced analysis system to support the interaction between the customer and the salesperson. First, acoustic data, visual data, bodily motion data, and text data are acquired using a receiving device. Acoustic data is obtained by recording the customer's voice using a microphone and converting it into text data using speech recognition technology. Visual data is obtained by capturing the customer's facial expressions and gestures with a camera and analyzing it using image recognition technology. Bodily motion data is obtained by detecting the customer's movements with a motion sensor and processing it with a motion analysis algorithm.

[0518] The server processes this data in real time and performs detailed analysis based on the characteristics of each data point using deep learning models. By integrating these analysis results and evaluating customer emotions and intentions, the most effective customer service strategy is generated. The generated strategy is presented to the sales staff and visually fed back using smart glasses or display devices.

[0519] Furthermore, for particularly complex customer requests, adaptive learning of deep learning models can be utilized to improve analysis accuracy. This system also incorporates data anonymization, providing strong protection for customer privacy.

[0520] For example, if analysis of customer voices detects that the customer is excited, the system will advise the salesperson to "explain things carefully to the customer and take your time guiding them." As an example of a prompt, it will generate a question such as, "The customer's tone of voice has started to change. What emotions could this indicate?" to draw the salesperson's attention.

[0521] Thus, the present invention makes it possible to increase customer satisfaction and expand sales opportunities.

[0522] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0523] Step 1:

[0524] The user uses the device's camera and microphone to acquire customer audio data, visual data, body movement data, and conversation data. Audio and visual information are captured as input. This collects basic data on customer interactions.

[0525] Step 2:

[0526] To analyze the received data, the server first removes noise from the audio data and extracts the necessary frames from the visual data. The input here is the original audio and video, and the output is clear audio data with noise removed and the important visual frames. This preprocessing step prepares the data for analysis.

[0527] Step 3:

[0528] The server inputs pre-processed data into a deep learning model to perform sentiment and intent analysis. Using the audio data, a natural language processing model generates text data. Pre-processed audio and text are provided as input, and the analysis results are obtained as output. This makes it possible to evaluate the customer's emotional state and conversational intent.

[0529] Step 4:

[0530] The server integrates the analysis results and generates an optimal customer service strategy by utilizing information obtained from multiple data sources. The input consists of results from voice analysis, facial recognition, and gesture analysis, which are then integrated and outputted using a generated AI model to determine the best course of action. This results in a unified and effective strategy based on all the data.

[0531] Step 5:

[0532] The generated customer service strategy is visually fed back to smart glasses or displays connected to the terminal. The server takes detailed information about the strategy as input and provides the user with visualized advice to facilitate operation. This allows the user to quickly understand and implement specific customer service policies.

[0533] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0534] The multimodal communication analysis system according to the present invention analyzes user emotions using various nonverbal information and provides an effective communication strategy. The embodiments for carrying out this invention are described below.

[0535] First, the user powers on the device and prepares it to use the system. The device is connected to a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting bodily movement information. The user uses these devices to collect the necessary data.

[0536] The device collects the user's voice as acoustic information in real time and captures facial expressions as visual information via a camera. It also monitors the user's movements using sensors to obtain information about their physical actions. Furthermore, the user provides the content of conversations and negotiations by inputting text information into the device.

[0537] The server analyzes the received audio, visual, bodily movement, and text information. First, it performs preprocessing to remove noise from the audio information and divide the visual information into frames. Next, it applies an emotion analysis engine to the audio and visual information. For the audio information, it recognizes the user's emotions by analyzing the tone, tempo, and volume of the voice. Similarly, it analyzes facial expressions from the visual information to determine the user's emotional state.

[0538] After the analysis is complete, the server integrates this information to evaluate the user's overall emotional state and negotiation situation. Based on this integrated data, the server generates optimal advice. The generated advice is presented to the user in an intuitively understandable format. Specifically, it is presented through graphs, text, and voice assistance displayed on the device screen.

[0539] For example, in a business negotiation, if the server detects that the other party is experiencing stress, it may suggest a break to ease the tension or offer advice on adjusting the content of the conversation. This allows the user to implement effective communication strategies in real time.

[0540] Furthermore, this invention anonymizes received information to strictly protect the privacy of user data. This allows users to use the system with peace of mind in a privacy-protected environment.

[0541] The following describes the processing flow.

[0542] Step 1:

[0543] The user powers on the device and configures it to start the multimodal communication analysis system. The device verifies that the microphone, camera, and sensors are functioning correctly and prepares to collect data.

[0544] Step 2:

[0545] The device uses a microphone to record the user's speech as acoustic information and a camera to record the user's facial expressions as visual information. It monitors the user's body movements through sensors and captures actual gestures and postures. In addition, it collects text information when the user inputs text.

[0546] Step 3:

[0547] The server receives raw data transmitted from the terminal in real time. The received data includes acoustic information, visual information, bodily movement information, and textual information.

[0548] Step 4:

[0549] The server preprocesses the received data. Acoustic information undergoes noise filtering, and visual information is divided into frames for image analysis. Furthermore, text data is processed using natural language processing techniques to convert it into a format that is easy to analyze.

[0550] Step 5:

[0551] The server inputs pre-processed data into the emotion analysis engine. It analyzes acoustic information to identify emotions from voice tone and tempo, and uses facial recognition technology to estimate emotional states from facial expressions.

[0552] Step 6:

[0553] The server integrates the analysis results. It combines the results from acoustic, visual, bodily movement, and textual information to assess the user's overall emotional state and understand the context of negotiations and conversations.

[0554] Step 7:

[0555] The server generates advice based on integrated analysis results. It uses deep learning models as needed to propose situation-sensitive communication strategies in real time.

[0556] Step 8:

[0557] The server sends the generated advice to the terminal. The terminal presents the advice to the user using visual and audio elements. Visual information includes graphs and text displayed on the dashboard, with audio guides supplementing necessary explanations.

[0558] Step 9:

[0559] Users use the provided advice and analysis results to choose their next action in actual negotiations and conversations. This allows users to achieve better communication outcomes.

[0560] (Example 2)

[0561] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] Conventional communication analysis systems have struggled to effectively integrate information from different modalities and accurately grasp users' emotional states. Furthermore, privacy concerns exist, necessitating secure data handling. It is essential to address these issues and provide a more effective and secure communication support system.

[0563] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0564] In this invention, the server includes data acquisition means for acquiring acoustic data, visual data, motion data, and text data; data preprocessing means for preprocessing the various data acquired by the data acquisition means; and data analysis means for analyzing the data preprocessed by the data preprocessing means. This enables the integrated analysis of information from different modalities, allowing for accurate understanding of the user's emotional state and secure information provision that takes privacy into consideration.

[0565] "Acoustic data" refers to information composed of the user's voice and other sounds, and serves as the basis for analyzing emotions and states through the characteristics of sound.

[0566] "Visual data" refers to information obtained through the user's facial expressions and movements, and is used to visually capture their emotional state and reactions.

[0567] "Motion data" refers to information obtained from the user's body movements and posture, and serves as material for judging nonverbal emotional expressions and intentions.

[0568] "Text data" refers to text information entered by users and is used to clarify the content and intent of conversations.

[0569] "Data acquisition means" refers to functions and devices for collecting acoustic, visual, motion, and textual data, and forms the foundation for acquiring system input information.

[0570] "Data preprocessing means" refers to the processing performed to convert acquired raw data into a format that is easy to analyze, and includes processes such as noise reduction and frame splitting.

[0571] "Data analysis means" refers to a function for analyzing the user's emotional state and reactions based on pre-processed data, and it evaluates information from each modality to derive results.

[0572] "Data integration means" refers to a function that integrates analyzed information to perform a comprehensive evaluation, and is used to determine the user's overall emotional state.

[0573] The "proposal generation method" refers to a function that creates advice and suggestions for users based on integrated analysis results, and is intended to support communication strategies.

[0574] "Information presentation means" refers to functions that visually or audibly convey generated suggestions or advice to the user, providing information in a way that is easy for the user to understand.

[0575] The system according to the present invention is designed to analyze multifaceted user data and provide effective communication strategies. This system acquires and analyzes acoustic, visual, motor, and written information to understand the user's emotional state.

[0576] The user starts using the system by activating the terminal. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting motion information. These hardware devices allow the terminal to acquire the necessary data in real time. Acoustic data is collected through the user's voice, visual data is captured using the camera to capture facial expressions, and motion data is collected from body movements by sensors. The user also inputs text information into the terminal to provide conversation content and other information.

[0577] The server first preprocesses the information received from the terminal to improve data quality. Specifically, it removes noise from audio data and divides visual data into frames. An emotion analysis engine is then applied to this preprocessed data, which analyzes the tone and tempo of speech, facial expressions, etc., to identify the user's emotional state and response.

[0578] Based on the integrated analysis results, the server uses a generative AI model to create optimal suggestions. These suggestions are sent to the terminal and provided to the user through information presentation tools. Users can receive advice in an intuitively understandable format through graphs, text, and voice assistance displayed on the screen.

[0579] For example, if the system analyzes that the other party is experiencing stress during a business negotiation, the server will generate advice suggesting an appropriate break for the user. A possible prompt might be, "What is a recommended break time when the other party is experiencing stress during a business negotiation?"

[0580] This invention further enhances the protection of personal information by performing de-identification processing to safeguard the confidentiality of user data. This mechanism provides users with an environment in which they can use the system with peace of mind.

[0581] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0582] Step 1:

[0583] The user activates the device and prepares to begin collecting acoustic, visual, motion, and text data. The device uses its microphone, camera, and sensors to acquire acoustic, visual, and motion data as input, while simultaneously collecting text data entered by the user. This prepares raw data on various aspects of communication.

[0584] Step 2:

[0585] The device removes noise from the acquired acoustic data and outputs clear audio data that preserves the tone and tempo of the voice. For visual data, the input from the camera is divided frame by frame and processed so that facial expressions can be clearly identified. For motion data, the raw data obtained from the sensors is organized based on the time series of motion to prepare accurate motion information. This preprocessing yields clean, structured data necessary for subsequent analysis steps.

[0586] Step 3:

[0587] The server receives pre-processed data and applies an emotion analysis engine. For acoustic data, it analyzes voice tone, tempo, and volume; for visual data, it analyzes facial expressions; and from motion data, it extracts nonverbal emotional indicators of the user. These analysis results are output individually to provide detailed insights into the user's emotional state.

[0588] Step 4:

[0589] The server integrates the analysis results and uses a generative AI model to comprehensively evaluate the information from each modality. This allows it to understand the user's overall emotional state and reactions, and prepare advice based on that information. The generated advice is then adjusted to the user's specific situation, enabling the implementation of the optimal communication strategy.

[0590] Step 5:

[0591] The server sends the final advice as data to the terminal and provides it to the user through information presentation tools. The terminal outputs the advice using graphs, text, and voice assistance, presenting the information in a way that the user can understand and act upon in real time. In addition, the raw data entered is anonymized during processing, contributing to the protection of user privacy.

[0592] (Application Example 2)

[0593] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0594] In modern society, security is increasingly important, but conventional security systems struggle to accurately grasp human emotional states and therefore fail to adequately detect potential risks. In this context, there is a need for technologies that can provide effective communication strategies in real time and improve security.

[0595] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0596] In this invention, the server includes a device for receiving acoustic information, visual information, and operational information; a processing unit for pre-processing the information acquired by the device; and an analysis unit for analyzing the information processed by the processing unit and determining the emotional state. This makes it possible to detect abnormalities early, generate and provide feedback based on security evaluations.

[0597] "Acoustic information" refers to audio data obtained from a subject and the results of its analysis.

[0598] "Visual information" refers to video data of the subject and the results of its analysis.

[0599] "Motion information" refers to data and analysis results related to the movements and posture of the subject's body.

[0600] "Receiving device" refers to a hardware device or equipment with related functions for acquiring acoustic information, visual information, and operational information.

[0601] A "pre-processing unit" refers to a computer system or group of programs that removes noise from received information and converts it into a format suitable for analysis.

[0602] The "analysis unit" is the part that performs algorithms and calculations to identify specific emotional states or patterns from pre-processed data.

[0603] A "security evaluation device" is a device or system that determines risks and detects anomalies based on analyzed information.

[0604] A "generation mechanism" refers to a means or process for generating user feedback based on information obtained from security evaluation equipment.

[0605] An "output device" is hardware or an interface that provides the generated feedback to the user.

[0606] To implement this invention, it is necessary to construct a system equipped with devices for receiving acoustic information, visual information, and motion information. First, for acoustic information, a high-sensitivity microphone is required to capture the target's voice in real time. Next, for visual information, a high-resolution camera is used to capture the target's face and movements. Furthermore, to acquire motion information, a motion sensor is incorporated to analyze the target's body movements.

[0607] Upon receiving this data, the server first uses a preprocessor to remove noise and normalize the data, and then the analysis unit determines the emotional state. The analysis unit is equipped with speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API). This allows the server to determine emotions from the tone and tempo of the acoustic information, changes in facial expressions from visual information, and body movements.

[0608] Based on these results, the security assessment device detects anomalies. For example, in airport security, if a visitor shows signs of potential stress or anxiety, the security assessment device recognizes this and generates corresponding feedback. The generation mechanism constructs appropriate advice or warnings based on the detected information and presents them to the user through the output device.

[0609] The following is a concrete example of a prompt statement generated using a generative AI model:

[0610] "Using multimodal data, we assess visitors' emotional states and determine if they are experiencing increased anxiety or stress. If an anomaly is detected, security personnel are notified in real time."

[0611] This system makes it possible to detect and respond to risks more effectively than conventional security systems.

[0612] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0613] Step 1:

[0614] The user activates the device and prepares various sensors to collect acoustic, visual, and motion information. At this stage, the user's voice is captured by the microphone, and facial images are collected by the camera. Motion sensors also record the user's movements in real time. This data is then transmitted from the device to the server.

[0615] Step 2:

[0616] The server preprocesses the received data. Specifically, it denoises the audio data and breaks down the visual data into frames. Here, the input is raw audio, visual, and motion information, and the output is data that has been formatted to be analyzable.

[0617] Step 3:

[0618] The server analyzes the pre-processed data using speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API) to identify the user's emotional state. The input is pre-processed acoustic and visual information, and the output is an indicator of the user's emotional state. In this step, emotions are analyzed based on factors such as voice tone and facial expression changes.

[0619] Step 4:

[0620] The server uses a security evaluation device to detect anomalies based on the analysis results. The input here is the result obtained from emotion analysis, and the output is an evaluation value indicating the degree of risk. This process generates a warning, especially if high levels of stress or anxiety are detected.

[0621] Step 5:

[0622] The server generates anomaly-based feedback using a generation mechanism. The input is an anomaly detection evaluation value, and the output is specific advice or warning messages presented to the user. The feedback is evaluated in real time and shaped appropriately.

[0623] Step 6:

[0624] The device provides feedback to the user via an output device. This feedback is presented through the device's screen or audio output. For example, a warning message such as, "Your stress level is high. Additional checks may be needed," might be displayed.

[0625] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0626] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0628] [Fourth Embodiment]

[0629] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0630] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0631] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0633] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0635] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0636] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0637] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0638] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0639] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0640] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0641] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0642] The multimodal communication analysis AI agent according to the present invention is a system that enables a comprehensive understanding of nonverbal elements in negotiation and dialogue situations by receiving, analyzing, integrating various types of information and providing advice on optimal strategies. Its embodiments are described below.

[0643] First, the user activates the system and collects the necessary data through the terminal. The terminal picks up participants' voices as acoustic information using a microphone and captures video as visual information using a camera. It also obtains physical movement information by detecting the user's movements with sensors. Finally, it obtains text information by inputting the content of negotiations and conversations as text.

[0644] The server receives this information in real time and performs effective preprocessing. Specifically, it removes noise from acoustic data and extracts necessary frames from visual data. It also performs text analysis on text data using natural language processing techniques. Body movement information is converted into a format that can be analyzed by motion recognition algorithms.

[0645] During the analysis phase, the server analyzes each piece of pre-processed information based on its specific characteristics. Voice analysis evaluates participants' emotions and intentions, while facial expression analysis captures subtle facial changes. Gesture analysis tracks hand and body movements and extracts important ones. Text analysis understands the tone and intent of speech to grasp the overall picture.

[0646] Next, the server comprehensively evaluates these analysis results to capture the participants' emotional states and the atmosphere of the negotiation in a unified manner. Based on this integrated analysis, the server generates appropriate advice. The advice generation utilizes deep learning techniques and past pattern analysis results to propose the optimal strategy for the specific situation the user is facing.

[0647] The generated advice is provided to the user through the device. The user receives visual feedback on the device screen and can refer to audio guides. This allows the user to immediately obtain useful information and effectively advance negotiations and discussions.

[0648] Furthermore, the server ensures that all data is anonymized and securely managed to protect user privacy. This allows users to use the system with peace of mind.

[0649] The following describes the processing flow.

[0650] Step 1:

[0651] The user activates the terminal and starts the multimodal communication analysis system. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting body movement information.

[0652] Step 2:

[0653] The device begins data collection upon receiving a start command from the user. It collects acoustic information using a microphone, acquires visual information using a camera, and records body movement information using sensors. For text information, the user inputs the text into the device.

[0654] Step 3:

[0655] The server receives raw data transmitted from the terminal. This received data includes acoustic, visual, bodily motion, and textual information.

[0656] Step 4:

[0657] The server preprocesses the received data. Noise reduction is performed on acoustic data, and frame division is carried out on visual data. Body movement data is formatted into a unified sensor data format, and text data is converted into a form suitable for language analysis using natural language processing techniques.

[0658] Step 5:

[0659] The server analyzes the pre-processed data. It performs emotion analysis on acoustic information and facial expression analysis on visual information. It performs gesture analysis on bodily movement information and analyzes intent and context on written information.

[0660] Step 6:

[0661] The server integrates the results of each analysis. Based on the integrated information, it evaluates the overall communication situation and understands the participants' emotions and circumstances.

[0662] Step 7:

[0663] The server generates advice from integrated data. It uses deep learning and analysis of historical patterns to suggest optimal negotiation strategies and actions for the user.

[0664] Step 8:

[0665] The server sends the generated advice to the terminal. The terminal presents the results, including the advice, to the user in visual and auditory formats.

[0666] Step 9:

[0667] Users receive advice and analysis results displayed on their devices and decide on their next actions based on them. This enables users to communicate effectively in real time.

[0668] (Example 1)

[0669] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] Nonverbal elements play a crucial role in modern negotiations and dialogues. However, comprehensively understanding and effectively analyzing this diverse information is challenging. In particular, there is a need to accurately grasp participants' emotional states and the atmosphere of negotiations in real time and provide appropriate advice based on that understanding, but existing technologies are unable to meet this challenge.

[0671] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0672] In this invention, the server includes acquisition means for acquiring audio data, video data, body movement data, and text data; a data processing mechanism for preprocessing the data obtained by the acquisition means; and data analysis means for analyzing the preprocessed data provided by the data processing mechanism. This enables centralized analysis of diverse information and the generation of optimal advice.

[0673] "Acquisition means" refers to a mechanism for collecting audio data, video data, body movement data, and text data and providing them to a server.

[0674] A "data processing mechanism" is a device or method that organizes data obtained through acquisition means and prepares it in a format that can be analyzed.

[0675] A "data analysis tool" is a system or process for analyzing pre-processed data and extracting and evaluating specific information based on its content.

[0676] An "analysis integration tool" is a function that integrates information obtained from individual data analyses and evaluates it from a more holistic perspective.

[0677] A "content generation device" is a mechanism that creates suggestions and advice for users based on integrated analysis results.

[0678] An "information transmission system" is a means of conveying the generated proposal content to users visually or audibly.

[0679] The "evaluation function" is a function within the analysis and integration system that determines the characteristics of emotional states and reactions based on the obtained data.

[0680] "Anonymization" is a technique or method that removes or conceals personally identifiable information from data in order to protect privacy.

[0681] This multimodal communication analysis system is designed to enable users to effectively acquire and analyze information in negotiations and dialogues, and to obtain optimal advice. The system has the functionality to collect and analyze audio data, video data, bodily motion data, and text data.

[0682] The user activates the system and collects necessary data using a terminal. The terminal has a built-in microphone, camera, and motion sensor, and uses this hardware to acquire voice, video, and body movement information. In addition, the content of negotiations and conversations can be entered as text data via a keyboard or other input devices.

[0683] The server receives data from terminals in real time and performs preprocessing using a data processing mechanism. Preprocessing includes noise reduction for audio data, frame extraction for video data, and natural language processing for text data. This prepares the data for analysis.

[0684] The analyzed data is integrated and analyzed using server-generated AI models to understand the emotional state of participants and the atmosphere of negotiations. Based on these analysis results, optimal advice is generated. The advice is provided as the most appropriate response for the current situation, based on a model that has learned past data patterns.

[0685] The device provides feedback to the user with the generated advice. Visual feedback is displayed on the screen, and audio guidance is used as needed. For example, it can show specific negotiation strategies or provide the results of an analysis of the user's emotional state.

[0686] Regarding privacy protection, the server anonymizes all data and securely manages user information. This protection feature allows users to use the system with peace of mind.

[0687] A concrete example would be a situation where, "a client is concerned about the price of a new product, and we need advice on how to emphasize the product's added value instead of lowering the price." An example of a prompt in this case would be:

[0688] Question for the AI ​​agent: If a client expresses concerns about pricing, how can I emphasize the added value of the product without lowering the price?

[0689] In this way, the system provides immediate and effective support for the challenges that users face.

[0690] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0691] Step 1:

[0692] The user activates the device and collects the data necessary for negotiations and conversations. Inputs include acoustic data acquired by the device's microphone, video data captured by the camera, body movement data detected by sensors, and text data entered by the user. Output is a composite data stream sent to the server.

[0693] Step 2:

[0694] The server receives a composite data stream from the terminal and performs preprocessing using a data processing mechanism. Specific operations include noise reduction on audio data input, extraction of necessary frames from video data, and natural language processing for text data. The output of this processing is a dataset in a format suitable for analysis.

[0695] Step 3:

[0696] The server analyzes the pre-processed data using data analysis tools. Based on each data input, it analyzes emotional nuances from audio data, changes in facial expressions and gestures from video data, and intentions and tone from text data. The output of this step is the analysis results, such as emotional states.

[0697] Step 4:

[0698] The server integrates the analysis results obtained from the data analysis means using the analysis integration means. The analysis results are unified, summarizing the overall atmosphere of the negotiation and the emotional state of the participants. The output of this integration process is the integrated evaluation result.

[0699] Step 5:

[0700] The server generates optimal advice using a generative AI model based on the integrated evaluation results. Based on the integrated results, it devises countermeasures for specific situations the user faces and creates advice statements. The output is the specific advice provided to the user.

[0701] Step 6:

[0702] The terminal provides the user with advice from the server. The advice is displayed visually on the screen and accompanied by specific feedback actions explained by voice guidance. The final output of this step is easy-to-understand advice received by the user.

[0703] Step 7:

[0704] The server anonymizes the acquired data, ensuring secure information management while protecting privacy. Each data element is anonymized and kept inaccessible from the outside. The output of this process is an anonymized, secure data record.

[0705] (Application Example 1)

[0706] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] In many brick-and-mortar stores, sales staff often struggle to accurately understand customers' emotions and intentions, resulting in inadequate customer service. Furthermore, a lack of tools to enable sales staff to analyze customer reactions in real time and propose optimal service approaches poses challenges to improving customer satisfaction and maximizing sales opportunities.

[0708] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0709] In this invention, the server includes means for receiving acoustic data, visual data, bodily movement data, and text data; means for preprocessing the various data acquired by the receiving means; and means for analyzing the data preprocessed by the preprocessing means. This makes it possible to analyze emotional state and reaction data in real time during interaction with customers and propose the optimal customer service method.

[0710] "Audio data" refers to audio information acquired via audio input devices such as microphones, and includes audio information related to customer speech and conversations.

[0711] "Visual data" refers to video information acquired by video input devices such as cameras, and includes a series of visual information such as a customer's facial expressions and movements.

[0712] "Body movement data" refers to data that includes information about a person's body movements and posture obtained from acceleration sensors and motion sensors.

[0713] "Text data" refers to data that includes information obtained by converting speech into text information using speech recognition technology, and which includes information that is a transcription of customer utterances.

[0714] "Analysis results" refer to information obtained after analyzing acoustic data, visual data, bodily movement data, and text data, including evaluations of the customer's emotional state and intentions.

[0715] A "strategy generation tool" is a device or algorithm that performs a series of processes to generate an optimal communication strategy based on analysis results.

[0716] A "presentation means" is a means of showing a generated communication strategy to the user, and is a device that transmits information visually or audibly using displays and audio equipment.

[0717] "Optimization suggestion means" refers to a system or process that makes specific suggestions for improving customer service methods based on real-time customer feedback.

[0718] This shows an embodiment for carrying out the invention.

[0719] To realize this application example, the present invention is configured as follows: The server provides an advanced analysis system to support the interaction between the customer and the salesperson. First, acoustic data, visual data, bodily motion data, and text data are acquired using a receiving device. Acoustic data is obtained by recording the customer's voice using a microphone and converting it into text data using speech recognition technology. Visual data is obtained by capturing the customer's facial expressions and gestures with a camera and analyzing it using image recognition technology. Bodily motion data is obtained by detecting the customer's movements with a motion sensor and processing it with a motion analysis algorithm.

[0720] The server processes this data in real time and performs detailed analysis based on the characteristics of each data point using deep learning models. By integrating these analysis results and evaluating customer emotions and intentions, the most effective customer service strategy is generated. The generated strategy is presented to the sales staff and visually fed back using smart glasses or display devices.

[0721] Furthermore, for particularly complex customer requests, adaptive learning of deep learning models can be utilized to improve analysis accuracy. This system also incorporates data anonymization, providing strong protection for customer privacy.

[0722] For example, if analysis of customer voices detects that the customer is excited, the system will advise the salesperson to "explain things carefully to the customer and take your time guiding them." As an example of a prompt, it will generate a question such as, "The customer's tone of voice has started to change. What emotions could this indicate?" to draw the salesperson's attention.

[0723] Thus, the present invention makes it possible to increase customer satisfaction and expand sales opportunities.

[0724] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0725] Step 1:

[0726] The user uses the device's camera and microphone to acquire customer audio data, visual data, body movement data, and conversation data. Audio and visual information are captured as input. This collects basic data on customer interactions.

[0727] Step 2:

[0728] To analyze the received data, the server first removes noise from the audio data and extracts the necessary frames from the visual data. The input here is the original audio and video, and the output is clear audio data with noise removed and the important visual frames. This preprocessing step prepares the data for analysis.

[0729] Step 3:

[0730] The server inputs pre-processed data into a deep learning model to perform sentiment and intent analysis. Using the audio data, a natural language processing model generates text data. Pre-processed audio and text are provided as input, and the analysis results are obtained as output. This makes it possible to evaluate the customer's emotional state and conversational intent.

[0731] Step 4:

[0732] The server integrates the analysis results and generates an optimal customer service strategy by utilizing information obtained from multiple data sources. The input consists of results from voice analysis, facial recognition, and gesture analysis, which are then integrated and outputted using a generated AI model to determine the best course of action. This results in a unified and effective strategy based on all the data.

[0733] Step 5:

[0734] The generated customer service strategy is visually fed back to smart glasses or displays connected to the terminal. The server takes detailed information about the strategy as input and provides the user with visualized advice to facilitate operation. This allows the user to quickly understand and implement specific customer service policies.

[0735] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0736] The multimodal communication analysis system according to the present invention analyzes user emotions using various nonverbal information and provides an effective communication strategy. The embodiments for carrying out this invention are described below.

[0737] First, the user powers on the device and prepares it to use the system. The device is connected to a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting bodily movement information. The user uses these devices to collect the necessary data.

[0738] The device collects the user's voice as acoustic information in real time and captures facial expressions as visual information via a camera. It also monitors the user's movements using sensors to obtain information about their physical actions. Furthermore, the user provides the content of conversations and negotiations by inputting text information into the device.

[0739] The server analyzes the received audio, visual, bodily movement, and text information. First, it performs preprocessing to remove noise from the audio information and divide the visual information into frames. Next, it applies an emotion analysis engine to the audio and visual information. For the audio information, it recognizes the user's emotions by analyzing the tone, tempo, and volume of the voice. Similarly, it analyzes facial expressions from the visual information to determine the user's emotional state.

[0740] After the analysis is complete, the server integrates this information to evaluate the user's overall emotional state and negotiation situation. Based on this integrated data, the server generates optimal advice. The generated advice is presented to the user in an intuitively understandable format. Specifically, it is presented through graphs, text, and voice assistance displayed on the device screen.

[0741] For example, in a business negotiation, if the server detects that the other party is experiencing stress, it may suggest a break to ease the tension or offer advice on adjusting the content of the conversation. This allows the user to implement effective communication strategies in real time.

[0742] Furthermore, this invention anonymizes received information to strictly protect the privacy of user data. This allows users to use the system with peace of mind in a privacy-protected environment.

[0743] The following describes the processing flow.

[0744] Step 1:

[0745] The user powers on the device and configures it to start the multimodal communication analysis system. The device verifies that the microphone, camera, and sensors are functioning correctly and prepares to collect data.

[0746] Step 2:

[0747] The device uses a microphone to record the user's speech as acoustic information and a camera to record the user's facial expressions as visual information. It monitors the user's body movements through sensors and captures actual gestures and postures. In addition, it collects text information when the user inputs text.

[0748] Step 3:

[0749] The server receives raw data transmitted from the terminal in real time. The received data includes acoustic information, visual information, bodily movement information, and textual information.

[0750] Step 4:

[0751] The server preprocesses the received data. Acoustic information undergoes noise filtering, and visual information is divided into frames for image analysis. Furthermore, text data is processed using natural language processing techniques to convert it into a format that is easy to analyze.

[0752] Step 5:

[0753] The server inputs pre-processed data into the emotion analysis engine. It analyzes acoustic information to identify emotions from voice tone and tempo, and uses facial recognition technology to estimate emotional states from facial expressions.

[0754] Step 6:

[0755] The server integrates the analysis results. It combines the results from acoustic, visual, bodily movement, and textual information to assess the user's overall emotional state and understand the context of negotiations and conversations.

[0756] Step 7:

[0757] The server generates advice based on integrated analysis results. It uses deep learning models as needed to propose situation-sensitive communication strategies in real time.

[0758] Step 8:

[0759] The server sends the generated advice to the terminal. The terminal presents the advice to the user using visual and audio elements. Visual information includes graphs and text displayed on the dashboard, with audio guides supplementing necessary explanations.

[0760] Step 9:

[0761] Users use the provided advice and analysis results to choose their next action in actual negotiations and conversations. This allows users to achieve better communication outcomes.

[0762] (Example 2)

[0763] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0764] Conventional communication analysis systems have struggled to effectively integrate information from different modalities and accurately grasp users' emotional states. Furthermore, privacy concerns exist, necessitating secure data handling. It is essential to address these issues and provide a more effective and secure communication support system.

[0765] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0766] In this invention, the server includes data acquisition means for acquiring acoustic data, visual data, motion data, and text data; data preprocessing means for preprocessing the various data acquired by the data acquisition means; and data analysis means for analyzing the data preprocessed by the data preprocessing means. This enables the integrated analysis of information from different modalities, allowing for accurate understanding of the user's emotional state and secure information provision that takes privacy into consideration.

[0767] "Acoustic data" refers to information composed of the user's voice and other sounds, and serves as the basis for analyzing emotions and states through the characteristics of sound.

[0768] "Visual data" refers to information obtained through the user's facial expressions and movements, and is used to visually capture their emotional state and reactions.

[0769] "Motion data" refers to information obtained from the user's body movements and posture, and serves as material for judging nonverbal emotional expressions and intentions.

[0770] "Text data" refers to text information entered by users and is used to clarify the content and intent of conversations.

[0771] "Data acquisition means" refers to functions and devices for collecting acoustic, visual, motion, and textual data, and forms the foundation for acquiring system input information.

[0772] "Data preprocessing means" refers to the processing performed to convert acquired raw data into a format that is easy to analyze, and includes processes such as noise reduction and frame splitting.

[0773] "Data analysis means" refers to a function for analyzing the user's emotional state and reactions based on pre-processed data, and it evaluates information from each modality to derive results.

[0774] "Data integration means" refers to a function that integrates analyzed information to perform a comprehensive evaluation, and is used to determine the user's overall emotional state.

[0775] The "proposal generation method" refers to a function that creates advice and suggestions for users based on integrated analysis results, and is intended to support communication strategies.

[0776] "Information presentation means" refers to functions that visually or audibly convey generated suggestions or advice to the user, providing information in a way that is easy for the user to understand.

[0777] The system according to the present invention is designed to analyze multifaceted user data and provide effective communication strategies. This system acquires and analyzes acoustic, visual, motor, and written information to understand the user's emotional state.

[0778] The user starts using the system by activating the terminal. The terminal is equipped with a microphone for acquiring acoustic information, a camera for acquiring visual information, and sensors for collecting motion information. These hardware devices allow the terminal to acquire the necessary data in real time. Acoustic data is collected through the user's voice, visual data is captured using the camera to capture facial expressions, and motion data is collected from body movements by sensors. The user also inputs text information into the terminal to provide conversation content and other information.

[0779] The server first preprocesses the information received from the terminal to improve data quality. Specifically, it removes noise from audio data and divides visual data into frames. An emotion analysis engine is then applied to this preprocessed data, which analyzes the tone and tempo of speech, facial expressions, etc., to identify the user's emotional state and response.

[0780] Based on the integrated analysis results, the server uses a generative AI model to create optimal suggestions. These suggestions are sent to the terminal and provided to the user through information presentation tools. Users can receive advice in an intuitively understandable format through graphs, text, and voice assistance displayed on the screen.

[0781] For example, if the system analyzes that the other party is experiencing stress during a business negotiation, the server will generate advice suggesting an appropriate break for the user. A possible prompt might be, "What is a recommended break time when the other party is experiencing stress during a business negotiation?"

[0782] This invention further enhances the protection of personal information by performing de-identification processing to safeguard the confidentiality of user data. This mechanism provides users with an environment in which they can use the system with peace of mind.

[0783] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0784] Step 1:

[0785] The user activates the device and prepares to begin collecting acoustic, visual, motion, and text data. The device uses its microphone, camera, and sensors to acquire acoustic, visual, and motion data as input, while simultaneously collecting text data entered by the user. This prepares raw data on various aspects of communication.

[0786] Step 2:

[0787] The device removes noise from the acquired acoustic data and outputs clear audio data that preserves the tone and tempo of the voice. For visual data, the input from the camera is divided frame by frame and processed so that facial expressions can be clearly identified. For motion data, the raw data obtained from the sensors is organized based on the time series of motion to prepare accurate motion information. This preprocessing yields clean, structured data necessary for subsequent analysis steps.

[0788] Step 3:

[0789] The server receives pre-processed data and applies an emotion analysis engine. For acoustic data, it analyzes voice tone, tempo, and volume; for visual data, it analyzes facial expressions; and from motion data, it extracts nonverbal emotional indicators of the user. These analysis results are output individually to provide detailed insights into the user's emotional state.

[0790] Step 4:

[0791] The server integrates the analysis results and uses a generative AI model to comprehensively evaluate the information from each modality. This allows it to understand the user's overall emotional state and reactions, and prepare advice based on that information. The generated advice is then adjusted to the user's specific situation, enabling the implementation of the optimal communication strategy.

[0792] Step 5:

[0793] The server sends the final advice as data to the terminal and provides it to the user through information presentation tools. The terminal outputs the advice using graphs, text, and voice assistance, presenting the information in a way that the user can understand and act upon in real time. In addition, the raw data entered is anonymized during processing, contributing to the protection of user privacy.

[0794] (Application Example 2)

[0795] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0796] In modern society, security is increasingly important, but conventional security systems struggle to accurately grasp human emotional states and therefore fail to adequately detect potential risks. In this context, there is a need for technologies that can provide effective communication strategies in real time and improve security.

[0797] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0798] In this invention, the server includes a device for receiving acoustic information, visual information, and operational information; a processing unit for pre-processing the information acquired by the device; and an analysis unit for analyzing the information processed by the processing unit and determining the emotional state. This makes it possible to detect abnormalities early, generate and provide feedback based on security evaluations.

[0799] "Acoustic information" refers to audio data obtained from a subject and the results of its analysis.

[0800] "Visual information" refers to video data of the subject and the results of its analysis.

[0801] "Motion information" refers to data and analysis results related to the movements and posture of the subject's body.

[0802] "Receiving device" refers to a hardware device or equipment with related functions for acquiring acoustic information, visual information, and operational information.

[0803] A "pre-processing unit" refers to a computer system or group of programs that removes noise from received information and converts it into a format suitable for analysis.

[0804] The "analysis unit" is the part that performs algorithms and calculations to identify specific emotional states or patterns from pre-processed data.

[0805] A "security evaluation device" is a device or system that determines risks and detects anomalies based on analyzed information.

[0806] A "generation mechanism" refers to a means or process for generating user feedback based on information obtained from security evaluation equipment.

[0807] An "output device" is hardware or an interface that provides the generated feedback to the user.

[0808] To implement this invention, it is necessary to construct a system equipped with devices for receiving acoustic information, visual information, and motion information. First, for acoustic information, a high-sensitivity microphone is required to capture the target's voice in real time. Next, for visual information, a high-resolution camera is used to capture the target's face and movements. Furthermore, to acquire motion information, a motion sensor is incorporated to analyze the target's body movements.

[0809] Upon receiving this data, the server first uses a preprocessor to remove noise and normalize the data, and then the analysis unit determines the emotional state. The analysis unit is equipped with speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API). This allows the server to determine emotions from the tone and tempo of the acoustic information, changes in facial expressions from visual information, and body movements.

[0810] Based on these results, the security assessment device detects anomalies. For example, in airport security, if a visitor shows signs of potential stress or anxiety, the security assessment device recognizes this and generates corresponding feedback. The generation mechanism constructs appropriate advice or warnings based on the detected information and presents them to the user through the output device.

[0811] The following is a concrete example of a prompt statement generated using a generative AI model:

[0812] "Using multimodal data, we assess visitors' emotional states and determine if they are experiencing increased anxiety or stress. If an anomaly is detected, security personnel are notified in real time."

[0813] This system makes it possible to detect and respond to risks more effectively than conventional security systems.

[0814] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0815] Step 1:

[0816] The user activates the device and prepares various sensors to collect acoustic, visual, and motion information. At this stage, the user's voice is captured by the microphone, and facial images are collected by the camera. Motion sensors also record the user's movements in real time. This data is then transmitted from the device to the server.

[0817] Step 2:

[0818] The server preprocesses the received data. Specifically, it denoises the audio data and breaks down the visual data into frames. Here, the input is raw audio, visual, and motion information, and the output is data that has been formatted to be analyzable.

[0819] Step 3:

[0820] The server analyzes the pre-processed data using speech processing software (e.g., Google Speech-to-Text) and image recognition software (e.g., Microsoft Azure Face API) to identify the user's emotional state. The input is pre-processed acoustic and visual information, and the output is an indicator of the user's emotional state. In this step, emotions are analyzed based on factors such as voice tone and facial expression changes.

[0821] Step 4:

[0822] The server uses a security evaluation device to detect anomalies based on the analysis results. The input here is the result obtained from emotion analysis, and the output is an evaluation value indicating the degree of risk. This process generates a warning, especially if high levels of stress or anxiety are detected.

[0823] Step 5:

[0824] The server generates anomaly-based feedback using a generation mechanism. The input is an anomaly detection evaluation value, and the output is specific advice or warning messages presented to the user. The feedback is evaluated in real time and shaped appropriately.

[0825] Step 6:

[0826] The device provides feedback to the user via an output device. This feedback is presented through the device's screen or audio output. For example, a warning message such as, "Your stress level is high. Additional checks may be needed," might be displayed.

[0827] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0828] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0829] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0830] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0831] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0832] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0833] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0834] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0835] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0836] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0837] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0838] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0839] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0840] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0841] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0842] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0843] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0844] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0845] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0846] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0847] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0848] The following is further disclosed regarding the embodiments described above.

[0849] (Claim 1)

[0850] Receiving means for receiving acoustic information, visual information, bodily movement information, and textual information,

[0851] A preprocessing means for preprocessing various information acquired by the receiving means,

[0852] An analysis means for analyzing the information preprocessed by the aforementioned preprocessing means,

[0853] Integration means for integrating the analysis results obtained by the analysis means,

[0854] An advice generation means that generates advice based on the analysis results integrated by the aforementioned integration means,

[0855] An output means for providing the aforementioned advice to the user,

[0856] A system that includes this.

[0857] (Claim 2)

[0858] The system according to claim 1, characterized in that the integration means comprises an evaluation means for evaluating emotional state and reaction information obtained from the analysis results.

[0859] (Claim 3)

[0860] The system according to claim 1, wherein the receiving means performs anonymization processing to protect the privacy of user data.

[0861] "Example 1"

[0862] (Claim 1)

[0863] Acquisition means for acquiring audio data, video data, body movement data, and text data,

[0864] A data processing mechanism for preprocessing the data obtained by the acquisition means,

[0865] A data analysis means for analyzing pre-processed data provided by the data processing mechanism,

[0866] An analysis integration means for integrating the analysis information obtained by the data analysis means,

[0867] A content generation device that generates proposed content based on the aforementioned analysis and integration means,

[0868] An information transmission system for communicating the above-mentioned proposal to users,

[0869] A system that includes this.

[0870] (Claim 2)

[0871] The system according to claim 1, characterized in that the analysis integration means has an evaluation function that takes into account the emotional state and reaction characteristics extracted from the analysis information.

[0872] (Claim 3)

[0873] The system according to claim 1, characterized in that the acquisition means performs data anonymization processing for the purpose of protecting user information.

[0874] "Application Example 1"

[0875] (Claim 1)

[0876] A receiving means for receiving acoustic data, visual data, body movement data, and text data,

[0877] A preprocessing means for preprocessing various data acquired by the receiving means,

[0878] An analysis means for analyzing the data preprocessed by the aforementioned preprocessing means,

[0879] Integration means for integrating the analysis results obtained by the analysis means,

[0880] A strategy generation means that generates a strategy based on the analysis results integrated by the aforementioned integration means,

[0881] A means for presenting the aforementioned strategy to the user,

[0882] An optimization suggestion method that proposes the optimization of communication methods based on results analyzed in real time during customer-salesperson conversations,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, characterized in that the integration means includes an evaluation means that evaluates emotional state and reaction data obtained from the analysis results and proposes an appropriate customer service method according to the sales situation.

[0886] (Claim 3)

[0887] The system according to claim 1, characterized in that the receiving means performs anonymization processing to protect the privacy of user data.

[0888] "Example 2 of combining an emotion engine"

[0889] (Claim 1)

[0890] A data acquisition means for acquiring acoustic data, visual data, motion data, and text data,

[0891] A data preprocessing means for preprocessing various data acquired by the data acquisition means,

[0892] A data analysis means for analyzing the data preprocessed by the data preprocessing means,

[0893] A data integration means for integrating the analysis results obtained by the data analysis means,

[0894] A proposal generation means that generates proposals based on the analysis results integrated by the data integration means,

[0895] Information presentation means for providing the aforementioned proposal to users,

[0896] A system that includes this.

[0897] (Claim 2)

[0898] The system according to claim 1, characterized in that the data integration means includes an evaluation function for evaluating emotional state and reaction information obtained from the analysis results.

[0899] (Claim 3)

[0900] The system according to claim 1, characterized in that the data acquisition means performs despecification processing to protect the confidentiality of user information.

[0901] "Application example 2 when combining with an emotional engine"

[0902] (Claim 1)

[0903] A device that receives acoustic information, visual information, and motion information,

[0904] A processing unit that preprocesses the information acquired by the aforementioned device,

[0905] An analysis unit analyzes the information processed by the aforementioned processing unit and determines the emotional state,

[0906] A security evaluation device that detects abnormalities based on the emotional state obtained by the analysis unit,

[0907] A generation mechanism that generates feedback based on anomalies detected by the security evaluation device,

[0908] The output device that provides the aforementioned feedback,

[0909] A system that includes this.

[0910] (Claim 2)

[0911] The system according to claim 1, characterized in that the generation mechanism comprises a method for analyzing stress levels from non-vocal signs.

[0912] (Claim 3)

[0913] The system according to claim 1, characterized in that the device performs a specific removal process for the purpose of protecting personal information. [Explanation of Symbols]

[0914] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Receiving means for receiving acoustic information, visual information, bodily movement information, and textual information, A preprocessing means for preprocessing various information acquired by the receiving means, An analysis means for analyzing the information preprocessed by the aforementioned preprocessing means, Integration means for integrating the analysis results obtained by the analysis means, An advice generation means that generates advice based on the analysis results integrated by the aforementioned integration means, An output means for providing the aforementioned advice to the user, A system that includes this.

2. The system according to claim 1, characterized in that the integration means comprises an evaluation means for evaluating emotional state and reaction information obtained from the analysis results.

3. The system according to claim 1, characterized in that the receiving means performs anonymization processing to protect the privacy of user data.

Images