system

Abstract

We provide the system. [Solution] Means for acquiring camera images, Means for acquiring audio data, A means of acquiring heart rate data, A means of sending data to a server, A means of analyzing the received data, A means for displaying the analysis results, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional contract system, it is difficult for consumers to judge the truth or falsehood of the words and actions of salespersons. Especially in the case of high-value transactions or transactions that require specialized knowledge, consumers are not guaranteed to obtain accurate information, so there are risks. If a salesperson deliberately provides false information, consumers may suffer significant economic losses. The purpose of this invention is to determine the truth or falsehood of the words and actions of salespersons in real time and provide consumers with highly reliable information.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for acquiring camera images, means for acquiring audio data, means for acquiring heart rate data, means for transmitting data to a server, means for analyzing the received data, and means for displaying the analysis results. In particular, it includes means for pre-processing, means for applying a face recognition algorithm, means for applying a voice recognition algorithm, means for synchronizing heart rate data over time, means for analyzing using a facial expression analysis model and a voice analysis model, and means for analyzing heart rate data. As a result, the system provides a way to determine the authenticity of a salesperson's words and actions in real time and display the determination results to consumers, thereby reducing consumer risk and enabling them to conduct transactions with peace of mind.

[0006] "Camera footage" refers to image data obtained from a device that converts and records visual information from the real world into digital data.

[0007] "Audio data" refers to data recorded by converting the waveform of sound into a digital signal.

[0008] "Heart rate data" refers to data that measures the user's heartbeat and shows the heart rate over a certain period of time.

[0009] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0010] "Preprocessing" refers to the process of organizing data and converting it into the required format before performing analysis.

[0011] A "face recognition algorithm" is a computational method for detecting human faces in digital images and analyzing their features.

[0012] A "speech recognition algorithm" is a computational method for converting words and sounds from audio data into text data.

[0013] A "facial expression analysis model" is a computational model that analyzes facial expressions and recognizes their movements and emotions.

[0014] The "voice analysis model" is a computational model for analyzing the characteristics of voice data and recognizing its changes and characteristics.

[0015] "Synchronizing on the time axis" is a process of aligning different types of data to the same time reference.

[0016] "Real-time" refers to a state where analysis and processing are performed almost simultaneously when data is generated.

[0017] "True or false" is the judgment result of whether a statement or information is true or false.

Brief Explanation of Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. This system acquires camera video, audio data, and heart rate data, transmits them to a server for analysis, and displays the analysis results in real time to the user wearing smart glasses.

[0040] Device (smart glasses) side

[0041] First, the user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, and microphone, and these devices are activated to begin data collection. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[0042] Server side

[0043] The server receives camera video, audio data, and heart rate data sent from the smart glasses. The received data is first pre-processed. This pre-processing includes dividing the video data into frames and applying a face recognition algorithm to detect the salesperson's face. The audio data is converted into text data using a speech recognition algorithm, and the heart rate data is synchronized over time.

[0044] Next, the server analyzes the pre-processed data. Specifically, it uses a facial expression analysis model to analyze the subtle movements of the salesperson's face and extract characteristics of stress and tension. It also uses a voice analysis model to analyze the characteristics of the voice data (tone, pitch, speed, etc.) and detects abnormal fluctuations from the heart rate data. By integrating these analysis results, it calculates the probability that the salesperson is lying.

[0045] The analysis results are calculated as specific percentages, generating messages such as "There is a 70% chance you are lying." The server sends this feedback message to the smart glasses in real time, notifying the user of the results.

[0046] User side

[0047] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0048] Specific example

[0049] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[0050] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk.

[0051] The following describes the processing flow.

[0052] Device (smart glasses) side

[0053] Step 1: Data Acquisition

[0054] The smart glasses activate the camera, heart rate sensor, and microphone. The camera captures the salesperson's image at 30 frames per second, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation.

[0055] Step 2: Send to the server

[0056] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, and audio data are transferred to the server via Wi-Fi or Bluetooth.

[0057] Server side

[0058] Step 3: Data Received

[0059] The server receives data sent from the smart glasses. All camera video data, audio data, and heart rate data are received using a secure communication protocol.

[0060] Step 4: Pretreatment

[0061] The server preprocesses the received data. Specifically, it performs the following processes:

[0062] The camera footage is divided into frames, and a face recognition algorithm is applied to detect the salesperson's face.

[0063] The audio data is converted to text using a speech recognition algorithm.

[0064] The heart rate data is synchronized with the video frames over time.

[0065] Step 5: Analysis

[0066] The server analyzes the pre-processed data. The specific operation is as follows:

[0067] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople to extract indicators of stress and tension.

[0068] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[0069] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[0070] Step 6: Truth Determination

[0071] The server integrates the results of facial expression analysis, voice analysis, and heart rate analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[0072] Step 7: Generate Feedback

[0073] The server generates a feedback message to notify the user based on the truth / false determination result. This message, in text format, might say something like, "There's a 70% chance you're lying."

[0074] Step 8: Submitting the analysis results

[0075] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[0076] Device (smart glasses) side

[0077] Step 9: Display User Interface

[0078] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[0079] User side

[0080] Step 10: Checking Feedback

[0081] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[0082] Step 11: Contract Decision

[0083] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[0084] Through this series of steps, a system will be created that allows consumers to verify the veracity of the salesperson's words and actions in real time, enabling them to proceed with the contract with confidence.

[0085] (Example 1)

[0086] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] In sales activities, it is essential to determine the veracity of a salesperson's words and actions in real time and provide consumers with reliable information. However, currently, it is difficult to instantly judge subtle psychological states such as a salesperson's stress or dishonesty. Unless this problem is solved, there is a risk that consumers will make important decisions based on inaccurate information.

[0088] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0089] In this invention, the server includes means for dividing camera images into frames, means for applying a face recognition algorithm, means for applying a speech recognition algorithm that converts audio data into text data, and means for synchronizing heart rate data over time. This makes it possible to analyze the words and actions of salespeople in real time and to quickly and accurately determine their authenticity.

[0090] "Means of acquiring camera footage" refers to devices or methods for recording the actions of salespeople in real time during sales activities.

[0091] "Means for acquiring audio data" refers to devices and methods for recording conversations between salespeople and consumers and collecting audio information for analysis.

[0092] "Means of acquiring heart rate data" refers to devices and methods for measuring a consumer's heart rate in real time and monitoring their stress and psychological state.

[0093] "Means of sending data to a server" refers to communication technologies and protocols used to send camera video, audio data, and heart rate data to a server.

[0094] "Means for pre-processing received data" refers to processing techniques used on the server side to convert various types of data received into a format that can be analyzed.

[0095] "Means for analyzing pre-processed data" refers to methods and devices for determining the veracity of a salesperson's statements and actions by applying various analytical algorithms to pre-processed data.

[0096] "Means of displaying analysis results" refers to displays and notification systems that communicate analysis results to consumers in real time.

[0097] "Methods for dividing camera footage into frames" refers to techniques or methods for breaking down continuous video data into individual still images (frames).

[0098] "Means for applying a face recognition algorithm" refers to algorithms and methods for executing them to detect and identify faces from acquired video data.

[0099] "Means of applying speech recognition algorithms that convert audio data into text data" refers to technology that analyzes audio data and converts its content into text format.

[0100] "Methods for synchronizing heart rate data over time" refers to technologies and methods for aligning heart rate data with other data (such as video or audio) on the same time axis.

[0101] "Methods of analysis using facial expression analysis models" refer to algorithms and technologies that analyze subtle facial movements to infer emotions and psychological states.

[0102] "Methods for analyzing tone, pitch, and speed using speech analysis models" refers to algorithms and technologies for analyzing the characteristics of speech data, such as tone, pitch, and speaking speed.

[0103] "Means for analyzing heart rate data" refers to methods or devices for analyzing acquired heart rate data and determining the physiological response of consumers.

[0104] "Methods for integrating multiple analysis results to calculate the probability that a salesperson is lying" refers to algorithms and technologies that integrate the results of facial expression analysis, voice analysis, and heart rate analysis to make an overall judgment.

[0105] "Means of notifying users of analysis results in real time" refers to communication technologies and display systems that immediately convey the results of the analysis to the user.

[0106] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. The system acquires camera video, audio data, and heart rate data, and sends this data to a server for analysis. The analysis results are displayed in real time to the user wearing smart glasses.

[0107] Device (smart glasses) side

[0108] The user first puts on the smart glasses. The smart glasses include the following devices:

[0109] Camera: Captures real-time footage of the salesperson.

[0110] Heart rate sensor: Monitors the user's heart rate.

[0111] Mike: Record the conversation with the salesman.

[0112] This data is transmitted to a server using Wi-Fi or Bluetooth. For example, while a salesperson is giving a product presentation, this video, audio, and heart rate data is collected in real time.

[0113] Server side

[0114] The server receives camera video, audio data, and heart rate data sent from the smart glasses. Specifically, it uses the following hardware and software.

[0115] Preprocessing of video data: The camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of the salespeople.

[0116] Audio data preprocessing: Convert audio data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[0117] Preprocessing of heart rate data: Synchronize heart rate data with video and audio data over time.

[0118] Next, the preprocessed data is analyzed. Specific analysis includes the following:

[0119] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[0120] A speech analysis model (e.g., Praat) is used to analyze characteristics such as tone, pitch, and speed.

[0121] Heart rate data analysis: Detects abnormal fluctuations.

[0122] These analysis results are combined to calculate the probability that the salesperson is lying. The results are displayed as a percentage, such as "there is a 70% probability that they are lying."

[0123] User side

[0124] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can ask additional questions or request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0125] Specific example

[0126] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and listens to the salesperson's explanation. During this time, the smart glasses' camera records the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes it in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use this information to ask the salesperson for clarification. This system allows users to reliably verify important contract details.

[0127] Example of a prompt

[0128] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[0129] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0130] Step 1: Data Collection

[0131] When a user puts on the smart glasses, the device simultaneously activates the camera, heart rate sensor, and microphone to begin collecting data. Inputs include the user's actions and their interaction with the salesperson. The camera captures the salesperson's video in real time, the heart rate sensor measures the user's heart rate in real time, and the microphone records the conversation. Outputs include captured video data, recorded audio data, and measured heart rate data. For example, video and audio of the moment the salesperson is explaining the product's features, along with the user's heart rate, are collected.

[0132] Step 2: Data transmission

[0133] The device transmits collected video, audio, and heart rate data to the server using Wi-Fi or Bluetooth. The input consists of the various data collected in step 1. Once the data transmission is complete, the data is received by the server as output. Specifically, the communication module of the smart glasses packets the data and sends it to the server.

[0134] Step 3: Data Preprocessing

[0135] The server receives data sent from the terminal. Inputs include camera video, audio data, and heart rate data transmitted in the data transmission step. The received data is first pre-processed. Specifically, the server divides the video data into frames and uses face recognition algorithms such as OpenCV to detect the salesperson's face. Audio data is converted to text data using the Google Speech-to-Text API, etc. Heart rate data is synchronized over time. Outputs include video frames in which faces were detected, the converted audio data, and the synchronized heart rate data.

[0136] Step 4: Data Analysis

[0137] The server analyzes the pre-processed data. The inputs include face detection video frames, text data, and synchronized heart rate data, all pre-processed in step 3. Specifically, it performs the following analyses:

[0138] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[0139] A speech analysis model (e.g., Praat) is used to analyze speech characteristics such as tone, pitch, and speed.

[0140] It detects abnormal fluctuations from heart rate data.

[0141] These analysis results are integrated to calculate the probability that the salesperson is lying. The output will include each analysis result and the integrated probability value. For example, a conclusion such as "there is a 70% probability that the salesperson is lying" might be drawn.

[0142] Step 5: Result Generation

[0143] The server generates a specific message based on the analysis results. The input is the probability value, which is an integrated version of the analysis results obtained in step 4. The generated message will display a percentage, such as "There is a 70% chance that you are lying." This message is obtained as output. Specifically, the server executes a message generation script and converts it into a user-friendly display format.

[0144] Step 6: Notification of Results

[0145] The server sends the generated analysis result message to the terminal (smart glasses) in real time. The input is the message generated in step 5. The sent message is displayed on the smart glasses' display and notified to the user. The output is the message displayed on the smart glasses. Specifically, the server's notification system packets the message and sends it to the terminal.

[0146] Step 7: User response

[0147] The user interacts with the salesperson while referring to the analysis results displayed on the smart glasses. The input is the analysis results displayed on the smart glasses. Based on the analysis results, the user can ask additional questions or request more detailed explanations. The output provides the user with information to make a more confident decision. Specifically, the user can ask the salesperson follow-up questions to confirm details.

[0148] Example of a prompt

[0149] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[0150] (Application Example 1)

[0151] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0152] Currently, consumers have limited means to determine the veracity of information provided in sales activities in real time, making it difficult to ensure consumer trust. Furthermore, directly verifying the truthfulness of salespeople's statements and actions is also difficult, posing risks when making important decisions. Therefore, a system is needed that allows consumers to conduct transactions with confidence.

[0153] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0154] In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a speech recognition algorithm, and means for synchronizing heart rate data over time. This makes it possible to integrate video analysis, audio analysis, and heart rate analysis to determine reliability.

[0155] "Means for acquiring camera images" refers to a function that uses cameras mounted on smart glasses or other devices to acquire images of a target in real time.

[0156] "Means for acquiring audio data" refers to a function that uses microphones built into smart glasses or other devices to record conversational audio in real time and acquire it as data.

[0157] "Means for acquiring heart rate data" refers to a function that uses heart rate sensors installed in smart glasses or other devices to acquire the user's heart rate data in real time.

[0158] "Means of sending data to the server" refers to the function of sending acquired camera video, audio data, and heart rate data to the server via the network.

[0159] "Means for analyzing received data" refers to algorithms and modules used to analyze data sent to a server and extract necessary information.

[0160] "Means for displaying analysis results" refers to a function for displaying the analysis results on the display of a device that has received the results (e.g., smart glasses).

[0161] "A means of notifying users of analysis results in real time" refers to a function that notifies users of the results in real time immediately after the analysis is completed.

[0162] "Means of preprocessing" refers to functions that clean and adjust data in order to improve the quality of data sent to the server.

[0163] "Means for applying a facial recognition algorithm" refers to an algorithm that recognizes a person's face based on video data and extracts specific features.

[0164] "Means for applying speech recognition algorithms" refers to algorithms that analyze speech data, convert it into text, and further analyze the characteristics of the speech.

[0165] "Methods for synchronizing heart rate data over time" refers to a function that links heart rate data with other data (such as video or audio data) in a temporal manner for analysis.

[0166] The "method for determining reliability by integrating video analysis, audio analysis, and heart rate analysis" is a function that comprehensively evaluates the results of multiple data analyses and calculates the reliability of a salesperson's words and actions.

[0167] "Means for calculating reliability evaluation results as probabilities" refers to a function that calculates, numerically, the degree of reliability based on integrated data analysis results.

[0168] "Means for displaying feedback messages on the smart glasses' display" refers to a function for displaying feedback messages based on analysis results on the smart glasses' display.

[0169] This invention is a system that supports sales activities in physical stores and allows consumers to verify the reliability of the information they are provided with. This system uses smart glasses as the main terminal, collecting data using a camera, microphone, and heart rate sensor, and analyzing the collected data on a server to determine the veracity of the salesperson's words and actions in real time.

[0170] Hardware configuration

[0171] Device (smart glasses)

[0172] The user wears smart glasses, and data is collected using the built-in camera, microphone, and heart rate sensor. The camera captures video of the salesperson in real time, the microphone records conversations, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth.

[0173] server

[0174] The server is responsible for the following roles:

[0175] 1. Data reception: Receives camera video, audio data, and heart rate data transmitted from the smart glasses.

[0176] 2. Data preprocessing: Video data is divided into frames, and a face recognition algorithm is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm, and heart rate data is synchronized over time.

[0177] 3. Data Analysis: A facial expression analysis model is used to analyze the subtle facial movements of the salesperson and extract characteristics of stress and tension. A voice analysis model is used to analyze the characteristics of the voice data (tone, pitch, speed, etc.), and abnormal fluctuations are also detected from heart rate data. These analysis results are integrated to calculate the probability that the salesperson is lying.

[0178] 4. Sending Results: The analysis results are calculated as specific percentages, and messages such as "There is a 70% chance that you are lying" are generated. The server sends this feedback message to the smart glasses in real time to notify the user of the results.

[0179] Software Configuration

[0180] The following software is used on the server side:

[0181] OpenCV: Used for preprocessing video data and face recognition.

[0182] Speech analysis module: For converting speech data into text and further analyzing the characteristics of the speech.

[0183] Heart rate analysis module: For detecting abnormal fluctuations in heart rate data.

[0184] Integrated analysis module: To integrate the analysis results of video, audio, and heart rate data to determine reliability.

[0185] Examples

[0186] As a concrete example, consider a scenario in a physical store where an expensive product is being described. The user wears smart glasses and receives an explanation from a store employee. The system allows the user to see the analysis results displayed on the smart glasses' screen in real time. Specifically, if it displays "there is an 80% probability that the explanation is reliable," the user can make a decision with confidence based on that information.

[0187] Example of a prompt

[0188] Next, we will show a concrete example of a prompt statement.

[0189] "We want to develop a system that analyzes the veracity of a salesperson's statements in real time when they provide detailed product descriptions, thereby supporting the provision of highly reliable information. This system will analyze camera footage, audio data, and heart rate data, and display the results on smart glasses. For example, we want the system to be usable in a scenario like this: when purchasing an expensive home appliance, the user wears smart glasses while receiving an explanation from a salesperson and can verify its reliability in real time. We would appreciate your advice on system design using sensors and analysis algorithms."

[0190] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0191] Step 1:

[0192] The user puts on smart glasses. The smart glasses activate the camera, microphone, and heart rate sensor and begin collecting data. Specifically, the camera captures video of the salesperson in real time, the microphone records the conversation, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth. The inputs are camera video, audio data, and heart rate data, and these are output as integrated data packets.

[0193] Step 2:

[0194] The server receives data packets transmitted from the smart glasses. The received data is first pre-processed. Specifically, the video data is divided into frames, the audio data is divided into fixed time units, and the heart rate data is compared with a timestamp. The input is raw data of camera video, audio data, and heart rate data, and the output is a pre-processed dataset.

[0195] Step 3:

[0196] The server applies a face recognition algorithm to pre-processed video data to detect the faces of salespeople. Specifically, it uses OpenCV to extract the position and major feature points of faces for each frame. The input is pre-processed video data, and the output is video data with face detection information added.

[0197] Step 4:

[0198] The server applies a speech recognition algorithm to the audio data, converting it into text data while simultaneously analyzing the characteristics of the speech (tone, pitch, speed, etc.). Specifically, it uses a speech recognition engine with a generative AI model to generate text from the audio data and extract speech features at the same time. The input is pre-processed audio data, and the output consists of text data and speech characteristic data.

[0199] Step 5:

[0200] The server analyzes heart rate data. This includes monitoring heart rate variability in real time and detecting any abnormalities. Specifically, it uses a heart rate analysis algorithm to evaluate heart rate data and detect signs of stress or tension. The input is pre-processed heart rate data, and the output is the analyzed heart rate data.

[0201] Step 6:

[0202] The server integrates the results of video analysis, audio analysis, and heart rate analysis to determine the reliability of the salesperson's words and actions. Specifically, it integrates the analysis results of each data to quantify the probability that the salesperson is lying. The inputs are video data with facial detection information, text data, audio characteristic data, and analyzed heart rate data, and the reliability evaluation results are output.

[0203] Step 7:

[0204] The server sends the evaluation results to the smart glasses and notifies the user. Specifically, it generates a reliability evaluation result as a feedback message and displays it on the smart glasses in real time. The input is the reliability evaluation result, and the feedback message is output. Based on the analysis results displayed on the smart glasses' screen, the user can proceed with the conversation with the salesperson. If a message such as "There is a 70% chance they are lying" is displayed, the user can ask further questions and request a more detailed explanation.

[0205] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0206] This invention is a system that uses a user emotion engine to determine the veracity of a salesperson's words and actions in real time, providing consumers with highly reliable information. This allows consumers to proceed with transactions with confidence.

[0207] Device (smart glasses) side

[0208] The user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, microphone, and emotion engine, and these devices are activated to begin collecting data. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, the microphone records the conversation, and the emotion engine obtains emotional data from the user's voice tone and facial expressions. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[0209] Server side

[0210] The server receives camera video, audio data, heart rate data, and emotion data sent from the smart glasses. The received data is first pre-processed, which includes the following:

[0211] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the faces of the salespeople.

[0212] The audio data is converted into text data using a speech recognition algorithm.

[0213] Heart rate data is synchronized with video frames over time.

[0214] User sentiment data acquired by the sentiment engine is also preprocessed.

[0215] Next, the server analyzes the pre-processed data. The specific steps are as follows:

[0216] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople and extract indicators of stress and tension.

[0217] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[0218] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[0219] We analyze user emotion data and evaluate the user's emotional state.

[0220] These analysis results are integrated to calculate the probability that the salesperson is lying. The analysis results are calculated as a specific percentage, and a message such as "There is a 70% chance that they are lying" is generated. The server sends this feedback message to the smart glasses in real time to notify the user of the result.

[0221] User side

[0222] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0223] Specific example

[0224] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[0225] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk. Furthermore, by combining it with an emotion engine, a comprehensive analysis that also takes into account the user's emotional state can be performed, enabling even more accurate determination of truthfulness.

[0226] The following describes the processing flow.

[0227] Device (smart glasses) side

[0228] Step 1: Data Acquisition

[0229] The smart glasses activate a camera, heart rate sensor, microphone, and emotion engine. The camera captures video of the salesperson at 30 frames per second, the heart rate sensor monitors the user's heart rate, the microphone records conversations, and the emotion engine extracts emotional data from the user's voice tone and facial expressions.

[0230] Step 2: Send to the server

[0231] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, audio data, and emotion data are transferred to the server via Wi-Fi or Bluetooth.

[0232] Server side

[0233] Step 3: Data Received

[0234] The server receives data sent from the smart glasses. All data, including camera video, audio, heart rate, and emotion data, is received using a secure communication protocol.

[0235] Step 4: Pretreatment

[0236] The server preprocesses the received data. The specific steps are as follows:

[0237] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the salesperson's face.

[0238] The audio data is converted to text using a speech recognition algorithm.

[0239] The heart rate data is synchronized with the video frame and timeline.

[0240] The emotional data is formatted appropriately to make it easier to analyze.

[0241] Step 5: Facial Expression Analysis

[0242] The server inputs pre-processed video data into a facial expression analysis model to analyze the subtle movements of the salesperson's face. It then extracts features that serve as indicators of stress and tension.

[0243] Step 6: Voice Analysis

[0244] The server inputs audio data into a speech analysis model, which analyzes speech characteristics such as tone, pitch, and speed. It also detects signs of stress and tension.

[0245] Step 7: Heart Rate Analysis

[0246] The server analyzes heart rate data and detects abnormal fluctuations such as sudden increases or irregularities in the user's heart rate.

[0247] Step 8: Emotion Analysis

[0248] The server analyzes pre-processed emotional data to evaluate the user's emotional state. It identifies emotional responses from the user's tone of voice and facial expressions.

[0249] Step 9: Comprehensive analysis and truth / false determination

[0250] The server integrates the results of facial expression analysis, voice analysis, heart rate analysis, and emotion analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[0251] Step 10: Generate Feedback

[0252] Based on the truth / false determination result, the server generates a feedback message to notify the user. This message, in text format, might state something like, "There is a 70% chance you are lying."

[0253] Step 11: Submitting the analysis results

[0254] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[0255] Device (smart glasses) side

[0256] Step 12: Display User Interface

[0257] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[0258] User side

[0259] Step 13: Checking Feedback

[0260] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[0261] Step 14: Decision on the contract

[0262] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[0263] Through the above series of steps, a system will be realized that allows consumers to verify the veracity of salespeople's words and actions in real time, enabling them to proceed with contracts with confidence. Furthermore, the introduction of an emotion engine will enable comprehensive analysis that also takes into account the user's emotional state, resulting in even more accurate determination of truthfulness.

[0264] (Example 2)

[0265] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0266] In traditional communication between consumers and salespeople, it was difficult to accurately determine the truthfulness of the salesperson's words and actions, resulting in a high risk of consumers suffering disadvantages. This problem prevented consumers from proceeding with transactions with confidence, especially in financial transactions and high-value contracts. Furthermore, traditional systems struggled to perform comprehensive analysis that took into account the user's emotional state, resulting in low accuracy in determining truthfulness.

[0267] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0268] In this invention, the server includes means for acquiring the user's biometric data, such as heart rate data; means for acquiring the user's emotional state; means for transmitting the data to the server via a communication network; means for pre-processing the received data; and means for integrating the analysis results of the pre-processed data and determining its truthfulness. This makes it possible to quickly and accurately determine the truthfulness of a salesperson's words and actions, allowing consumers to proceed with transactions with peace of mind. Furthermore, by performing a comprehensive analysis that also takes the user's emotional state into consideration, it becomes possible to determine truthfulness with even greater accuracy.

[0269] "Camera footage" refers to video data acquired by a camera installed in a wearable device worn by the user.

[0270] "Voice data" refers to audio information acquired by a microphone installed in a wearable device worn by the user.

[0271] "Heart rate data" refers to data obtained by measuring the user's heart rate, which is a biometric piece of information.

[0272] "Emotional state" refers to the psychological state of a user, obtained by analyzing their voice tone, facial expressions, and other biometric information.

[0273] A "communication network" refers to the infrastructure used to send and receive data, and this includes Wi-Fi, Bluetooth, and other similar technologies.

[0274] "Preprocessing" refers to a series of processes used to convert acquired raw data into an analyzable format, and includes data splitting, synchronization, and normalization.

[0275] "Analysis" is the process of evaluating the characteristics and relationships of pre-processed data using specific algorithms and models.

[0276] "True or false determination" means calculating the probability that a salesman's words and actions are true or false based on the analyzed data.

[0277] "Feedback" refers to the means of providing information for notifying the user of the analysis results, which includes visual displays, voice notifications, etc.

[0278] "Integration" refers to the process of aggregating different types of pre-processed and analyzed data (videos, voices, heartbeats, emotions, etc.) into one and making a comprehensive judgment.

[0279] The present invention is a system for enabling a user to proceed with a transaction with confidence in a conversation with a salesman. This system uses a wearable terminal (e.g., smart glasses) worn by the user and a server to determine in real time the truth or falsehood of the salesman's words and actions. The specific configuration and operation of the entire system will be described below.

[0280] Hardware Configuration

[0281] Terminal side (smart glasses)

[0282] Camera: Shoots videos in real time.

[0283] Microphone: Records conversations.

[0284] Heart rate sensor: Monitors the user's heart rate.

[0285] Emotion engine: Obtains emotion data from the tone of the user's voice and expressions.

[0286] Communication module: Has the function of transmitting data to the server via Wi-Fi or Bluetooth.

[0287] Server side

[0288] Data reception module: Receives data transmitted from smart glasses.

[0289] Preprocessing module: Preprocesses the received data.

[0290] Analysis module: Analyzes pre-processed data.

[0291] Feedback generation module: Generates feedback based on the analysis results.

[0292] Communication module: Sends the generated feedback to the smart glasses.

[0293] Data processing and data calculation

[0294] 1. The user puts on the smart glasses and begins the conversation.

[0295] 2. The device (smart glasses) collects camera video, audio data, heart rate data, and emotion data, and transmits them to the server in real time.

[0296] 3. The server divides the received video data into frames and applies a face recognition algorithm to detect the salesperson's face. For example, it uses computer vision technology such as OpenFace.

[0297] 4. The server converts the audio data into text data using a speech recognition algorithm. For example, it uses deep learning-based speech recognition technology such as Kaldi.

[0298] 5. The server uses techniques such as the Short-Term Fourier Transform to synchronize heart rate data and video frames over time.

[0299] 6. The server normalizes the emotional data and evaluates the user's psychological state as numerical data. It uses an emotional analysis model such as EmotionRecognitionNet.

[0300] 7. The server analyzes the preprocessed data and evaluates the characteristics of each data using an expression analysis model, a voice analysis model, and a heart rate analysis model.

[0301] 8. The server integrates the analysis results and calculates the probability that the salesman is lying using a Naive Bayes classifier.

[0302] 9. The server generates a specific feedback message such as "There is a 70% probability of lying" based on the probability and sends this to the smart glasses.

[0303] 10. The user proceeds with the conversation referring to the feedback displayed on the smart glasses. For example, based on the feedback, additional questions are asked to the salesman to confirm the authenticity of the information.

[0304] Specific example

[0305] As a specific example, consider the case of proceeding with a high-value housing loan contract. When the user wears smart glasses and listens to the explanation of the salesman, the camera of the smart glasses takes a video of the salesman, the heart rate sensor monitors the user's heart rate, the microphone records the conversation, and the emotion engine analyzes the tone and expression of the user's voice. These data are sent to the server in real time and analyzed immediately. As an analysis result, if it is determined that "there is an 80% probability that the salesman is lying about the content of the benefits", the user can request a reconfirmation from the salesman based on the warning message displayed on the smart glasses.

[0306] Example of prompt sentence

[0307] Please explain in detail the actions that the user should take when the smart glasses detect a lie with an 80% probability while the salesman is explaining the benefits in the scenario of a high-value housing loan contract.

[0308] The above describes a specific embodiment of the present invention. This system allows users to quickly and accurately determine the truthfulness of a salesperson's words and actions, enabling them to proceed with transactions with confidence. Furthermore, since the analysis also takes into account the user's emotional state, even more accurate determinations are possible.

[0309] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0310] Program processing flow

[0311] Step 1: Device startup and data collection

[0312] input:

[0313] The user puts on smart glasses.

[0314] process:

[0315] The camera on the device (smart glasses) activates and begins recording video in real time.

[0316] The microphone activates and begins recording the conversation.

[0317] The heart rate sensor begins measuring the user's heart rate.

[0318] The emotion engine begins analyzing the user's voice tone and facial expressions in real time.

[0319] output:

[0320] Collected video data, audio data, heart rate data, and emotion data.

[0321] Step 2: Send

[0322] input:

[0323] Collected video data, audio data, heart rate data, and emotion data.

[0324] process:

[0325] The device sends data to the server via Wi-Fi or Bluetooth.

[0326] The communication module compresses the data and sends it to the server via a secure network.

[0327] output:

[0328] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[0329] Step 3: Data Preprocessing

[0330] input:

[0331] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[0332] process:

[0333] The server divides the video data into frames and applies a face recognition algorithm (e.g., OpenFace) to detect the salesperson's face.

[0334] The server converts the audio data into text data using a speech recognition algorithm (e.g., Kaldi).

[0335] The server applies a short-term Fourier transform to synchronize the heart rate data with the video frame.

[0336] Emotional data is normalized, and the user's psychological state is evaluated as numerical data.

[0337] output:

[0338] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[0339] Step 4: Data Analysis

[0340] input:

[0341] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[0342] process:

[0343] The server uses a facial expression analysis model to analyze the subtle facial movements of the salesperson and extract indicators of stress and tension.

[0344] The server uses a speech analysis model to analyze the characteristics of the speech (tone, pitch, speed, etc.).

[0345] The server detects abnormal fluctuations from the heart rate data.

[0346] The server analyzes emotional data and evaluates the user's emotional state.

[0347] output:

[0348] Salesperson stress and tension indicators, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[0349] Step 5: Truth Determination and Feedback Generation

[0350] input:

[0351] Analyzed stress and tension indices, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[0352] process:

[0353] The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[0354] The server generates feedback messages expressing the truthfulness determination result as a specific percentage, such as "There is a 70% chance that they are lying."

[0355] output:

[0356] Feedback message (e.g., "There's a 70% chance you're lying").

[0357] Step 6: Displaying Feedback and User Interaction

[0358] input:

[0359] Feedback message.

[0360] process:

[0361] The device displays a feedback message on the smart glasses' display.

[0362] Voice notifications will also be provided as needed.

[0363] output:

[0364] A feedback message displayed to the user.

[0365] Step 7: User response

[0366] input:

[0367] The displayed feedback message.

[0368] process:

[0369] Users use the feedback messages to ask salespeople additional questions.

[0370] Users will use feedback messages to proceed with the dialogue more carefully.

[0371] output:

[0372] Asking additional questions to the salesperson and facilitating the conversation.

[0373] Through the above processing steps, the system can determine the veracity of the salesperson's statements in real time and provide feedback to the user. This allows the user to proceed with transactions with confidence.

[0374] (Application Example 2)

[0375] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0376] In sales activities, consumers often find it difficult to quickly and accurately determine the truthfulness of a salesperson's words and actions, making it difficult to evaluate the credibility of sales pitches and explanations, which can lead to disadvantages. Furthermore, it is not easy for consumers to understand their own emotional state and make calm judgments. In addition, there is a lack of tools to determine whether what a salesperson is saying is trustworthy, making it difficult to create an environment where consumers can proceed with transactions with peace of mind.

[0377] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a voice recognition algorithm, means for synchronizing heart rate data over time, and means for performing emotion analysis. This makes it possible to determine the veracity of the salesperson's words and actions in real time and provide consumers with highly reliable information. Furthermore, by analyzing emotion data, it is possible to perform a comprehensive analysis that also takes into account the consumer's emotional state and make even more accurate judgments. This makes it possible to provide consumers with an environment in which they can proceed with transactions with peace of mind.

[0378] "Camera footage" refers to visual information acquired through a camera.

[0379] "Audio data" refers to sound information acquired through a microphone.

[0380] "Heart rate data" refers to numerical information about the user's heart rate measured by a heart rate sensor.

[0381] "Emotional data" refers to data that indicates a user's emotional state, obtained from their tone of voice and facial expressions.

[0382] "Preprocessing" refers to the initial stages of processing that make data easier to analyze.

[0383] A "face recognition algorithm" refers to a computational method for automatically detecting human faces from video data.

[0384] A "speech recognition algorithm" refers to a computational method for converting speech data into text.

[0385] "Synchronizing across timeframes" refers to the process of adjusting different datasets to fit within the same timeframe.

[0386] "Emotional analysis" refers to a technology that analyzes acquired emotional data to evaluate the user's emotional state.

[0387] A "facial expression analysis model" refers to a computational model used to analyze emotions and psychological states from a person's facial expressions.

[0388] A "speech analysis model" refers to a computational model used to analyze the characteristics of speech (such as tone, pitch, and speed).

[0389] "Analysis results" refer to conclusions and evaluations obtained through the analysis of data.

[0390] "Real-time" refers to a state where the time from data acquisition to the display of analysis results is extremely short.

[0391] System Configuration

[0392] This invention provides a system that allows users to wear smart glasses to determine in real time whether a salesperson's words and actions are true or false, thereby providing consumers with reliable information. The system includes smart glasses and a server.

[0393] Device (smart glasses) side

[0394] The user wears smart glasses. The smart glasses have the following hardware and software:

[0395] Camera: Captures real-time footage of the salesperson.

[0396] Heart rate sensor: Monitors the user's heart rate.

[0397] Mike: Record the conversation.

[0398] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[0399] These devices transmit data to the server via Wi-Fi or Bluetooth.

[0400] Server side

[0401] The server receives data sent from the smart glasses and performs the following processing:

[0402] Preprocessing: Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted to text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames over time, and user emotion data obtained by the emotion engine is also preprocessed.

[0403] Data Analysis: We analyze the subtle facial movements of salespeople using facial expression analysis models (e.g., Dlib, DeepFace), and analyze the characteristics of voice data (tone, pitch, speed, etc.) using voice analysis models. We detect abnormal fluctuations (sudden increases and irregularities in heart rate) from heart rate data and analyze the user's emotional data.

[0404] Results Integration and Feedback: The analysis results are integrated to calculate the probability that the salesperson is lying. The result is calculated as a percentage, and a message such as "There is a 70% chance that they are lying" is generated. This feedback message is sent to the smart glasses in real time to notify the user of the result.

[0405] User side

[0406] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's words and actions and proceed with the contract with confidence.

[0407] Specific example

[0408] As a concrete example, consider the case of purchasing an expensive item (such as jewelry) and determining whether the salesperson's explanation is true. For instance, if a salesperson says, "This diamond is 100% pure and clean-cut," the "Trust Salesperson" application analyzes the information and displays "90% probability of being true" on the smart glasses' display. Based on this information, the customer can proceed with the purchase with confidence.

[0409] Example of a prompt

[0410] A salesperson explains, "This diamond is 100% pure and clean-cut." The customer is wearing smart glasses and using the "Trusted Salesperson" app, which analyzes the conversation, facial expressions, and voice characteristics in real time. The app indicates that there is a 99% probability that the salesperson is telling the truth. What does the customer do next?

[0411] By inputting this prompt into the AI ​​model, it is possible to support the dialogue between sales representatives and customers and facilitate reliable transactions.

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The device (smart glasses) acquires camera footage, audio data, heart rate data, and emotion data. The camera captures real-time video of the salesperson, and the microphone records conversations. The heart rate sensor monitors the user's heart rate, and the emotion engine acquires emotion data from the user's voice tone and facial expressions. This data is transmitted to a server via Wi-Fi or Bluetooth.

[0415] Inputs: Camera video, audio data, heart rate data, emotion data

[0416] Output: Data transmission from smart glasses to server

[0417] Step 2:

[0418] The server receives data transmitted from the terminal. The received data is first pre-processed. Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames in time, and emotion data is pre-processed.

[0419] Input: Raw data sent from the terminal

[0420] Output: Preprocessed data (frame-divided video, audio converted to text, synchronized heart rate data, preprocessed emotion data)

[0421] Step 3:

[0422] The server analyzes the pre-processed data. It uses facial expression analysis models (e.g., Dlib, DeepFace) to analyze the subtle facial movements of salespeople, and voice analysis models to analyze the characteristics of voice data (tone, pitch, speed, etc.). It detects abnormal fluctuations (sudden increases or irregularities in heart rate) from heart rate data and analyzes the user's emotional data.

[0423] Input: Preprocessed data

[0424] Output: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[0425] Step 4:

[0426] The server integrates the analysis results and calculates the probability that the salesperson is lying. A statistical model is used to comprehensively evaluate parameters obtained from different analyses in the integration of the analysis results. As a result, messages such as "There is a 70% probability that they are lying" are generated.

[0427] Input: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[0428] Output: Integrated analysis results (probability of lying)

[0429] Step 5:

[0430] The server transmits the integrated analysis results to the smart glasses in real time and notifies the user of the results. The notification is displayed as a message on the smart glasses' display.

[0431] Input: Integrated analysis results

[0432] Output: Feedback message displayed on smart glasses

[0433] Step 6:

[0434] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. This allows users to verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0435] Input: Feedback message displayed on smart glasses

[0436] Output: User actions (additional questions, contract progress)

[0437] Through the steps described above, the present invention provides an environment in which consumers can quickly and accurately determine the truthfulness of a salesperson's words and actions, and proceed with transactions with confidence.

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0441] [Second Embodiment]

[0442] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0443] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0448] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0449] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0450] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0453] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0454] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. This system acquires camera video, audio data, and heart rate data, transmits them to a server for analysis, and displays the analysis results in real time to the user wearing smart glasses.

[0455] Device (smart glasses) side

[0456] First, the user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, and microphone, and these devices are activated to begin data collection. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[0457] Server side

[0458] The server receives camera video, audio data, and heart rate data sent from the smart glasses. The received data is first pre-processed. This pre-processing includes dividing the video data into frames and applying a face recognition algorithm to detect the salesperson's face. The audio data is converted into text data using a speech recognition algorithm, and the heart rate data is synchronized over time.

[0459] Next, the server analyzes the pre-processed data. Specifically, it uses a facial expression analysis model to analyze the subtle movements of the salesperson's face and extract characteristics of stress and tension. It also uses a voice analysis model to analyze the characteristics of the voice data (tone, pitch, speed, etc.) and detects abnormal fluctuations from the heart rate data. By integrating these analysis results, it calculates the probability that the salesperson is lying.

[0460] The analysis results are calculated as specific percentages, generating messages such as "There is a 70% chance you are lying." The server sends this feedback message to the smart glasses in real time, notifying the user of the results.

[0461] User side

[0462] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0463] Specific example

[0464] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[0465] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk.

[0466] The following describes the processing flow.

[0467] Device (smart glasses) side

[0468] Step 1: Data Acquisition

[0469] The smart glasses activate the camera, heart rate sensor, and microphone. The camera captures the salesperson's image at 30 frames per second, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation.

[0470] Step 2: Send to the server

[0471] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, and audio data are transferred to the server via Wi-Fi or Bluetooth.

[0472] Server side

[0473] Step 3: Data Received

[0474] The server receives data sent from the smart glasses. All camera video data, audio data, and heart rate data are received using a secure communication protocol.

[0475] Step 4: Pretreatment

[0476] The server preprocesses the received data. Specifically, it performs the following processes:

[0477] The camera footage is divided into frames, and a face recognition algorithm is applied to detect the salesperson's face.

[0478] The audio data is converted to text using a speech recognition algorithm.

[0479] The heart rate data is synchronized with the video frames over time.

[0480] Step 5: Analysis

[0481] The server analyzes the pre-processed data. The specific operation is as follows:

[0482] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople to extract indicators of stress and tension.

[0483] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[0484] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[0485] Step 6: Truth Determination

[0486] The server integrates the results of facial expression analysis, voice analysis, and heart rate analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[0487] Step 7: Generate Feedback

[0488] The server generates a feedback message to notify the user based on the truth / false determination result. This message, in text format, might say something like, "There's a 70% chance you're lying."

[0489] Step 8: Submitting the analysis results

[0490] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[0491] Device (smart glasses) side

[0492] Step 9: Display User Interface

[0493] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[0494] User side

[0495] Step 10: Checking Feedback

[0496] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[0497] Step 11: Contract Decision

[0498] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[0499] Through this series of steps, a system will be created that allows consumers to verify the veracity of the salesperson's words and actions in real time, enabling them to proceed with the contract with confidence.

[0500] (Example 1)

[0501] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0502] In sales activities, it is essential to determine the veracity of a salesperson's words and actions in real time and provide consumers with reliable information. However, currently, it is difficult to instantly judge subtle psychological states such as a salesperson's stress or dishonesty. Unless this problem is solved, there is a risk that consumers will make important decisions based on inaccurate information.

[0503] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0504] In this invention, the server includes means for dividing camera images into frames, means for applying a face recognition algorithm, means for applying a speech recognition algorithm that converts audio data into text data, and means for synchronizing heart rate data over time. This makes it possible to analyze the words and actions of salespeople in real time and to quickly and accurately determine their authenticity.

[0505] "Means of acquiring camera footage" refers to devices or methods for recording the actions of salespeople in real time during sales activities.

[0506] "Means for acquiring audio data" refers to devices and methods for recording conversations between salespeople and consumers and collecting audio information for analysis.

[0507] "Means of acquiring heart rate data" refers to devices and methods for measuring a consumer's heart rate in real time and monitoring their stress and psychological state.

[0508] "Means of sending data to a server" refers to communication technologies and protocols used to send camera video, audio data, and heart rate data to a server.

[0509] "Means for pre-processing received data" refers to processing techniques used on the server side to convert various types of data received into a format that can be analyzed.

[0510] "Means for analyzing pre-processed data" refers to methods and devices for determining the veracity of a salesperson's statements and actions by applying various analytical algorithms to pre-processed data.

[0511] "Means of displaying analysis results" refers to displays and notification systems that communicate analysis results to consumers in real time.

[0512] "Methods for dividing camera footage into frames" refers to techniques or methods for breaking down continuous video data into individual still images (frames).

[0513] "Means for applying a face recognition algorithm" refers to algorithms and methods for executing them to detect and identify faces from acquired video data.

[0514] "Means of applying speech recognition algorithms that convert audio data into text data" refers to technology that analyzes audio data and converts its content into text format.

[0515] "Methods for synchronizing heart rate data over time" refers to technologies and methods for aligning heart rate data with other data (such as video or audio) on the same time axis.

[0516] "Methods of analysis using facial expression analysis models" refer to algorithms and technologies that analyze subtle facial movements to infer emotions and psychological states.

[0517] "Methods for analyzing tone, pitch, and speed using speech analysis models" refers to algorithms and technologies for analyzing the characteristics of speech data, such as tone, pitch, and speaking speed.

[0518] "Means for analyzing heart rate data" refers to methods or devices for analyzing acquired heart rate data and determining the physiological response of consumers.

[0519] "Methods for integrating multiple analysis results to calculate the probability that a salesperson is lying" refers to algorithms and technologies that integrate the results of facial expression analysis, voice analysis, and heart rate analysis to make an overall judgment.

[0520] "Means of notifying users of analysis results in real time" refers to communication technologies and display systems that immediately convey the results of the analysis to the user.

[0521] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. The system acquires camera video, audio data, and heart rate data, and sends this data to a server for analysis. The analysis results are displayed in real time to the user wearing smart glasses.

[0522] Device (smart glasses) side

[0523] The user first puts on the smart glasses. The smart glasses include the following devices:

[0524] Camera: Captures real-time footage of the salesperson.

[0525] Heart rate sensor: Monitors the user's heart rate.

[0526] Mike: Record the conversation with the salesman.

[0527] This data is transmitted to a server using Wi-Fi or Bluetooth. For example, while a salesperson is giving a product presentation, this video, audio, and heart rate data is collected in real time.

[0528] Server side

[0529] The server receives camera video, audio data, and heart rate data sent from the smart glasses. Specifically, it uses the following hardware and software.

[0530] Preprocessing of video data: The camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of the salespeople.

[0531] Audio data preprocessing: Convert audio data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[0532] Preprocessing of heart rate data: Synchronize heart rate data with video and audio data over time.

[0533] Next, the preprocessed data is analyzed. Specific analysis includes the following:

[0534] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[0535] A speech analysis model (e.g., Praat) is used to analyze characteristics such as tone, pitch, and speed.

[0536] Heart rate data analysis: Detects abnormal fluctuations.

[0537] These analysis results are combined to calculate the probability that the salesperson is lying. The results are displayed as a percentage, such as "there is a 70% probability that they are lying."

[0538] User side

[0539] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can ask additional questions or request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0540] Specific example

[0541] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and listens to the salesperson's explanation. During this time, the smart glasses' camera records the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes it in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use this information to ask the salesperson for clarification. This system allows users to reliably verify important contract details.

[0542] Example of a prompt

[0543] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[0544] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0545] Step 1: Data Collection

[0546] When a user puts on the smart glasses, the device simultaneously activates the camera, heart rate sensor, and microphone to begin collecting data. Inputs include the user's actions and their interaction with the salesperson. The camera captures the salesperson's video in real time, the heart rate sensor measures the user's heart rate in real time, and the microphone records the conversation. Outputs include captured video data, recorded audio data, and measured heart rate data. For example, video and audio of the moment the salesperson is explaining the product's features, along with the user's heart rate, are collected.

[0547] Step 2: Data transmission

[0548] The device transmits collected video, audio, and heart rate data to the server using Wi-Fi or Bluetooth. The input consists of the various data collected in step 1. Once the data transmission is complete, the data is received by the server as output. Specifically, the communication module of the smart glasses packets the data and sends it to the server.

[0549] Step 3: Data Preprocessing

[0550] The server receives data sent from the terminal. Inputs include camera video, audio data, and heart rate data transmitted in the data transmission step. The received data is first pre-processed. Specifically, the server divides the video data into frames and uses face recognition algorithms such as OpenCV to detect the salesperson's face. Audio data is converted to text data using the Google Speech-to-Text API, etc. Heart rate data is synchronized over time. Outputs include video frames in which faces were detected, the converted audio data, and the synchronized heart rate data.

[0551] Step 4: Data Analysis

[0552] The server analyzes the pre-processed data. The inputs include face detection video frames, text data, and synchronized heart rate data, all pre-processed in step 3. Specifically, it performs the following analyses:

[0553] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[0554] A speech analysis model (e.g., Praat) is used to analyze speech characteristics such as tone, pitch, and speed.

[0555] It detects abnormal fluctuations from heart rate data.

[0556] These analysis results are integrated to calculate the probability that the salesperson is lying. The output will include each analysis result and the integrated probability value. For example, a conclusion such as "there is a 70% probability that the salesperson is lying" might be drawn.

[0557] Step 5: Result Generation

[0558] The server generates a specific message based on the analysis results. The input is the probability value, which is an integrated version of the analysis results obtained in step 4. The generated message will display a percentage, such as "There is a 70% chance that you are lying." This message is obtained as output. Specifically, the server executes a message generation script and converts it into a user-friendly display format.

[0559] Step 6: Notification of Results

[0560] The server sends the generated analysis result message to the terminal (smart glasses) in real time. The input is the message generated in step 5. The sent message is displayed on the smart glasses' display and notified to the user. The output is the message displayed on the smart glasses. Specifically, the server's notification system packets the message and sends it to the terminal.

[0561] Step 7: User response

[0562] The user interacts with the salesperson while referring to the analysis results displayed on the smart glasses. The input is the analysis results displayed on the smart glasses. Based on the analysis results, the user can ask additional questions or request more detailed explanations. The output provides the user with information to make a more confident decision. Specifically, the user can ask the salesperson follow-up questions to confirm details.

[0563] Example of a prompt

[0564] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[0565] (Application Example 1)

[0566] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0567] Currently, consumers have limited means to determine the veracity of information provided in sales activities in real time, making it difficult to ensure consumer trust. Furthermore, directly verifying the truthfulness of salespeople's statements and actions is also difficult, posing risks when making important decisions. Therefore, a system is needed that allows consumers to conduct transactions with confidence.

[0568] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0569] In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a speech recognition algorithm, and means for synchronizing heart rate data over time. This makes it possible to integrate video analysis, audio analysis, and heart rate analysis to determine reliability.

[0570] "Means for acquiring camera images" refers to a function that uses cameras mounted on smart glasses or other devices to acquire images of a target in real time.

[0571] "Means for acquiring audio data" refers to a function that uses microphones built into smart glasses or other devices to record conversational audio in real time and acquire it as data.

[0572] "Means for acquiring heart rate data" refers to a function that uses heart rate sensors installed in smart glasses or other devices to acquire the user's heart rate data in real time.

[0573] "Means of sending data to the server" refers to the function of sending acquired camera video, audio data, and heart rate data to the server via the network.

[0574] "Means for analyzing received data" refers to algorithms and modules used to analyze data sent to a server and extract necessary information.

[0575] "Means for displaying analysis results" refers to a function for displaying the analysis results on the display of a device that has received the results (e.g., smart glasses).

[0576] "A means of notifying users of analysis results in real time" refers to a function that notifies users of the results in real time immediately after the analysis is completed.

[0577] "Means of preprocessing" refers to functions that clean and adjust data in order to improve the quality of data sent to the server.

[0578] "Means for applying a facial recognition algorithm" refers to an algorithm that recognizes a person's face based on video data and extracts specific features.

[0579] "Means for applying speech recognition algorithms" refers to algorithms that analyze speech data, convert it into text, and further analyze the characteristics of the speech.

[0580] "Methods for synchronizing heart rate data over time" refers to a function that links heart rate data with other data (such as video or audio data) in a temporal manner for analysis.

[0581] The "method for determining reliability by integrating video analysis, audio analysis, and heart rate analysis" is a function that comprehensively evaluates the results of multiple data analyses and calculates the reliability of a salesperson's words and actions.

[0582] "Means for calculating reliability evaluation results as probabilities" refers to a function that calculates, numerically, the degree of reliability based on integrated data analysis results.

[0583] "Means for displaying feedback messages on the smart glasses' display" refers to a function for displaying feedback messages based on analysis results on the smart glasses' display.

[0584] This invention is a system that supports sales activities in physical stores and allows consumers to verify the reliability of the information they are provided with. This system uses smart glasses as the main terminal, collecting data using a camera, microphone, and heart rate sensor, and analyzing the collected data on a server to determine the veracity of the salesperson's words and actions in real time.

[0585] Hardware configuration

[0586] Device (smart glasses)

[0587] The user wears smart glasses, and data is collected using the built-in camera, microphone, and heart rate sensor. The camera captures video of the salesperson in real time, the microphone records conversations, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth.

[0588] server

[0589] The server is responsible for the following roles:

[0590] 1. Data reception: Receives camera video, audio data, and heart rate data transmitted from the smart glasses.

[0591] 2. Data preprocessing: Video data is divided into frames, and a face recognition algorithm is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm, and heart rate data is synchronized over time.

[0592] 3. Data Analysis: A facial expression analysis model is used to analyze the subtle facial movements of the salesperson and extract characteristics of stress and tension. A voice analysis model is used to analyze the characteristics of the voice data (tone, pitch, speed, etc.), and abnormal fluctuations are also detected from heart rate data. These analysis results are integrated to calculate the probability that the salesperson is lying.

[0593] 4. Sending Results: The analysis results are calculated as specific percentages, and messages such as "There is a 70% chance that you are lying" are generated. The server sends this feedback message to the smart glasses in real time to notify the user of the results.

[0594] Software Configuration

[0595] The following software is used on the server side:

[0596] OpenCV: Used for preprocessing video data and face recognition.

[0597] Speech analysis module: For converting speech data into text and further analyzing the characteristics of the speech.

[0598] Heart rate analysis module: For detecting abnormal fluctuations in heart rate data.

[0599] Integrated analysis module: To integrate the analysis results of video, audio, and heart rate data to determine reliability.

[0600] Examples

[0601] As a concrete example, consider a scenario in a physical store where an expensive product is being described. The user wears smart glasses and receives an explanation from a store employee. The system allows the user to see the analysis results displayed on the smart glasses' screen in real time. Specifically, if it displays "there is an 80% probability that the explanation is reliable," the user can make a decision with confidence based on that information.

[0602] Example of a prompt

[0603] Next, we will show a concrete example of a prompt statement.

[0604] "We want to develop a system that analyzes the veracity of a salesperson's statements in real time when they provide detailed product descriptions, thereby supporting the provision of highly reliable information. This system will analyze camera footage, audio data, and heart rate data, and display the results on smart glasses. For example, we want the system to be usable in a scenario like this: when purchasing an expensive home appliance, the user wears smart glasses while receiving an explanation from a salesperson and can verify its reliability in real time. We would appreciate your advice on system design using sensors and analysis algorithms."

[0605] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0606] Step 1:

[0607] The user puts on smart glasses. The smart glasses activate the camera, microphone, and heart rate sensor and begin collecting data. Specifically, the camera captures video of the salesperson in real time, the microphone records the conversation, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth. The inputs are camera video, audio data, and heart rate data, and these are output as integrated data packets.

[0608] Step 2:

[0609] The server receives data packets transmitted from the smart glasses. The received data is first pre-processed. Specifically, the video data is divided into frames, the audio data is divided into fixed time units, and the heart rate data is compared with a timestamp. The input is raw data of camera video, audio data, and heart rate data, and the output is a pre-processed dataset.

[0610] Step 3:

[0611] The server applies a face recognition algorithm to pre-processed video data to detect the faces of salespeople. Specifically, it uses OpenCV to extract the position and major feature points of faces for each frame. The input is pre-processed video data, and the output is video data with face detection information added.

[0612] Step 4:

[0613] The server applies a speech recognition algorithm to the audio data, converting it into text data while simultaneously analyzing the characteristics of the speech (tone, pitch, speed, etc.). Specifically, it uses a speech recognition engine with a generative AI model to generate text from the audio data and extract speech features at the same time. The input is pre-processed audio data, and the output consists of text data and speech characteristic data.

[0614] Step 5:

[0615] The server analyzes heart rate data. This includes monitoring heart rate variability in real time and detecting any abnormalities. Specifically, it uses a heart rate analysis algorithm to evaluate heart rate data and detect signs of stress or tension. The input is pre-processed heart rate data, and the output is the analyzed heart rate data.

[0616] Step 6:

[0617] The server integrates the results of video analysis, audio analysis, and heart rate analysis to determine the reliability of the salesperson's words and actions. Specifically, it integrates the analysis results of each data to quantify the probability that the salesperson is lying. The inputs are video data with facial detection information, text data, audio characteristic data, and analyzed heart rate data, and the reliability evaluation results are output.

[0618] Step 7:

[0619] The server sends the evaluation results to the smart glasses and notifies the user. Specifically, it generates a reliability evaluation result as a feedback message and displays it on the smart glasses in real time. The input is the reliability evaluation result, and the feedback message is output. Based on the analysis results displayed on the smart glasses' screen, the user can proceed with the conversation with the salesperson. If a message such as "There is a 70% chance they are lying" is displayed, the user can ask further questions and request a more detailed explanation.

[0620] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0621] This invention is a system that uses a user emotion engine to determine the veracity of a salesperson's words and actions in real time, providing consumers with highly reliable information. This allows consumers to proceed with transactions with confidence.

[0622] Device (smart glasses) side

[0623] The user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, microphone, and emotion engine, and these devices are activated to begin collecting data. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, the microphone records the conversation, and the emotion engine obtains emotional data from the user's voice tone and facial expressions. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[0624] Server side

[0625] The server receives camera video, audio data, heart rate data, and emotion data sent from the smart glasses. The received data is first pre-processed, which includes the following:

[0626] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the faces of the salespeople.

[0627] The audio data is converted into text data using a speech recognition algorithm.

[0628] Heart rate data is synchronized with video frames over time.

[0629] User sentiment data acquired by the sentiment engine is also preprocessed.

[0630] Next, the server analyzes the pre-processed data. The specific steps are as follows:

[0631] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople and extract indicators of stress and tension.

[0632] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[0633] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[0634] We analyze user emotion data and evaluate the user's emotional state.

[0635] These analysis results are integrated to calculate the probability that the salesperson is lying. The analysis results are calculated as a specific percentage, and a message such as "There is a 70% chance that they are lying" is generated. The server sends this feedback message to the smart glasses in real time to notify the user of the result.

[0636] User side

[0637] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0638] Specific example

[0639] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[0640] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk. Furthermore, by combining it with an emotion engine, a comprehensive analysis that also takes into account the user's emotional state can be performed, enabling even more accurate determination of truthfulness.

[0641] The following describes the processing flow.

[0642] Device (smart glasses) side

[0643] Step 1: Data Acquisition

[0644] The smart glasses activate a camera, heart rate sensor, microphone, and emotion engine. The camera captures video of the salesperson at 30 frames per second, the heart rate sensor monitors the user's heart rate, the microphone records conversations, and the emotion engine extracts emotional data from the user's voice tone and facial expressions.

[0645] Step 2: Send to the server

[0646] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, audio data, and emotion data are transferred to the server via Wi-Fi or Bluetooth.

[0647] Server side

[0648] Step 3: Data Received

[0649] The server receives data sent from the smart glasses. All data, including camera video, audio, heart rate, and emotion data, is received using a secure communication protocol.

[0650] Step 4: Pretreatment

[0651] The server preprocesses the received data. The specific steps are as follows:

[0652] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the salesperson's face.

[0653] The audio data is converted to text using a speech recognition algorithm.

[0654] The heart rate data is synchronized with the video frame and timeline.

[0655] The emotional data is formatted appropriately to make it easier to analyze.

[0656] Step 5: Facial Expression Analysis

[0657] The server inputs pre-processed video data into a facial expression analysis model to analyze the subtle movements of the salesperson's face. It then extracts features that serve as indicators of stress and tension.

[0658] Step 6: Voice Analysis

[0659] The server inputs audio data into a speech analysis model, which analyzes speech characteristics such as tone, pitch, and speed. It also detects signs of stress and tension.

[0660] Step 7: Heart Rate Analysis

[0661] The server analyzes heart rate data and detects abnormal fluctuations such as sudden increases or irregularities in the user's heart rate.

[0662] Step 8: Emotion Analysis

[0663] The server analyzes pre-processed emotional data to evaluate the user's emotional state. It identifies emotional responses from the user's tone of voice and facial expressions.

[0664] Step 9: Comprehensive analysis and truth / false determination

[0665] The server integrates the results of facial expression analysis, voice analysis, heart rate analysis, and emotion analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[0666] Step 10: Generate Feedback

[0667] Based on the truth / false determination result, the server generates a feedback message to notify the user. This message, in text format, might state something like, "There is a 70% chance you are lying."

[0668] Step 11: Submitting the analysis results

[0669] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[0670] Device (smart glasses) side

[0671] Step 12: Display User Interface

[0672] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[0673] User side

[0674] Step 13: Checking Feedback

[0675] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[0676] Step 14: Decision on the contract

[0677] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[0678] Through the above series of steps, a system will be realized that allows consumers to verify the veracity of salespeople's words and actions in real time, enabling them to proceed with contracts with confidence. Furthermore, the introduction of an emotion engine will enable comprehensive analysis that also takes into account the user's emotional state, resulting in even more accurate determination of truthfulness.

[0679] (Example 2)

[0680] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0681] In traditional communication between consumers and salespeople, it was difficult to accurately determine the truthfulness of the salesperson's words and actions, resulting in a high risk of consumers suffering disadvantages. This problem prevented consumers from proceeding with transactions with confidence, especially in financial transactions and high-value contracts. Furthermore, traditional systems struggled to perform comprehensive analysis that took into account the user's emotional state, resulting in low accuracy in determining truthfulness.

[0682] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0683] In this invention, the server includes means for acquiring the user's biometric data, such as heart rate data; means for acquiring the user's emotional state; means for transmitting the data to the server via a communication network; means for pre-processing the received data; and means for integrating the analysis results of the pre-processed data and determining its truthfulness. This makes it possible to quickly and accurately determine the truthfulness of a salesperson's words and actions, allowing consumers to proceed with transactions with peace of mind. Furthermore, by performing a comprehensive analysis that also takes the user's emotional state into consideration, it becomes possible to determine truthfulness with even greater accuracy.

[0684] "Camera footage" refers to video data acquired by a camera installed in a wearable device worn by the user.

[0685] "Voice data" refers to audio information acquired by a microphone installed in a wearable device worn by the user.

[0686] "Heart rate data" refers to data obtained by measuring the user's heart rate, which is a biometric piece of information.

[0687] "Emotional state" refers to the psychological state of a user, obtained by analyzing their voice tone, facial expressions, and other biometric information.

[0688] A "communication network" refers to the infrastructure used to send and receive data, and this includes Wi-Fi, Bluetooth, and other similar technologies.

[0689] "Preprocessing" refers to a series of processes used to convert acquired raw data into an analyzable format, and includes data splitting, synchronization, and normalization.

[0690] "Analysis" is the process of evaluating the characteristics and relationships of pre-processed data using specific algorithms and models.

[0691] "Determining truthfulness" involves calculating the probability of whether a salesperson's statements or actions are true or false based on the analyzed data.

[0692] "Feedback" refers to a means of providing information to notify users of analysis results, and this includes visual displays and audio notifications.

[0693] "Integration" refers to the process of combining different types of pre-processed and analyzed data (such as video, audio, heart rate, and emotions) into a single, comprehensive analysis.

[0694] This invention is a system designed to allow users to proceed with transactions with confidence during interactions with sales representatives. This system uses a wearable device worn by the user (e.g., smart glasses) and a server to determine the veracity of the sales representative's statements in real time. The specific configuration and operation of the entire system are described below.

[0695] Hardware configuration

[0696] Device side (smart glasses)

[0697] Camera: Captures video in real time.

[0698] Mike: Record the conversation.

[0699] Heart rate sensor: Monitors the user's heart rate.

[0700] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[0701] Communication module: Has the function of sending data to a server via Wi-Fi or Bluetooth.

[0702] Server side

[0703] Data receiving module: Receives data transmitted from smart glasses.

[0704] Preprocessing module: Preprocesses the received data.

[0705] Analysis module: Analyzes pre-processed data.

[0706] Feedback generation module: Generates feedback based on the analysis results.

[0707] Communication module: Sends the generated feedback to the smart glasses.

[0708] Data processing and data calculation

[0709] 1. The user puts on the smart glasses and begins the conversation.

[0710] 2. The device (smart glasses) collects camera video, audio data, heart rate data, and emotion data, and transmits them to the server in real time.

[0711] 3. The server divides the received video data into frames and applies a face recognition algorithm to detect the salesperson's face. For example, it uses computer vision technology such as OpenFace.

[0712] 4. The server converts the audio data into text data using a speech recognition algorithm. For example, it uses deep learning-based speech recognition technology such as Kaldi.

[0713] 5. The server uses techniques such as the Short-Term Fourier Transform to synchronize heart rate data and video frames over time.

[0714] 6. The server normalizes the emotional data and evaluates the user's psychological state as numerical data. It uses an emotional analysis model such as EmotionRecognitionNet.

[0715] 7. The server analyzes the pre-processed data and evaluates the characteristics of each data point using facial expression analysis models, voice analysis models, and heart rate analysis models.

[0716] 8. The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[0717] 9. The server generates specific feedback messages based on probability, such as "There is a 70% chance you are lying," and sends these to the smart glasses.

[0718] 10. The user uses the feedback displayed on the smart glasses to guide the conversation. For example, they might ask the salesperson additional questions based on the feedback to verify the accuracy of the information.

[0719] Specific example

[0720] As a concrete example, consider the case of a high-value mortgage contract. When a user wears smart glasses and receives an explanation from a salesperson, the smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server in real time and analyzed immediately. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use the warning message displayed on the smart glasses to ask the salesperson for clarification again.

[0721] Example of a prompt

[0722] Please provide a detailed explanation of what a user should do if, during a high-value mortgage contract, smart glasses detect an 80% probability that the salesperson is lying while explaining the benefits.

[0723] The above describes a specific embodiment of the present invention. This system allows users to quickly and accurately determine the truthfulness of a salesperson's words and actions, enabling them to proceed with transactions with confidence. Furthermore, since the analysis also takes into account the user's emotional state, even more accurate determinations are possible.

[0724] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0725] Program processing flow

[0726] Step 1: Device startup and data collection

[0727] input:

[0728] The user puts on smart glasses.

[0729] process:

[0730] The camera on the device (smart glasses) activates and begins recording video in real time.

[0731] The microphone activates and begins recording the conversation.

[0732] The heart rate sensor begins measuring the user's heart rate.

[0733] The emotion engine begins analyzing the user's voice tone and facial expressions in real time.

[0734] output:

[0735] Collected video data, audio data, heart rate data, and emotion data.

[0736] Step 2: Send

[0737] input:

[0738] Collected video data, audio data, heart rate data, and emotion data.

[0739] process:

[0740] The device sends data to the server via Wi-Fi or Bluetooth.

[0741] The communication module compresses the data and sends it to the server via a secure network.

[0742] output:

[0743] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[0744] Step 3: Data Preprocessing

[0745] input:

[0746] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[0747] process:

[0748] The server divides the video data into frames and applies a face recognition algorithm (e.g., OpenFace) to detect the salesperson's face.

[0749] The server converts the audio data into text data using a speech recognition algorithm (e.g., Kaldi).

[0750] The server applies a short-term Fourier transform to synchronize the heart rate data with the video frame.

[0751] Emotional data is normalized, and the user's psychological state is evaluated as numerical data.

[0752] output:

[0753] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[0754] Step 4: Data Analysis

[0755] input:

[0756] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[0757] process:

[0758] The server uses a facial expression analysis model to analyze the subtle facial movements of the salesperson and extract indicators of stress and tension.

[0759] The server uses a speech analysis model to analyze the characteristics of the speech (tone, pitch, speed, etc.).

[0760] The server detects abnormal fluctuations from the heart rate data.

[0761] The server analyzes emotional data and evaluates the user's emotional state.

[0762] output:

[0763] Salesperson stress and tension indicators, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[0764] Step 5: Truth Determination and Feedback Generation

[0765] input:

[0766] Analyzed stress and tension indices, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[0767] process:

[0768] The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[0769] The server generates feedback messages expressing the truthfulness determination result as a specific percentage, such as "There is a 70% chance that they are lying."

[0770] output:

[0771] Feedback message (e.g., "There's a 70% chance you're lying").

[0772] Step 6: Displaying Feedback and User Interaction

[0773] input:

[0774] Feedback message.

[0775] process:

[0776] The device displays a feedback message on the smart glasses' display.

[0777] Voice notifications will also be provided as needed.

[0778] output:

[0779] A feedback message displayed to the user.

[0780] Step 7: User response

[0781] input:

[0782] The displayed feedback message.

[0783] process:

[0784] Users use the feedback messages to ask salespeople additional questions.

[0785] Users will use feedback messages to proceed with the dialogue more carefully.

[0786] output:

[0787] Asking additional questions to the salesperson and facilitating the conversation.

[0788] Through the above processing steps, the system can determine the veracity of the salesperson's statements in real time and provide feedback to the user. This allows the user to proceed with transactions with confidence.

[0789] (Application Example 2)

[0790] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0791] In sales activities, consumers often find it difficult to quickly and accurately determine the truthfulness of a salesperson's words and actions, making it difficult to evaluate the credibility of sales pitches and explanations, which can lead to disadvantages. Furthermore, it is not easy for consumers to understand their own emotional state and make calm judgments. In addition, there is a lack of tools to determine whether what a salesperson is saying is trustworthy, making it difficult to create an environment where consumers can proceed with transactions with peace of mind.

[0792] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a voice recognition algorithm, means for synchronizing heart rate data over time, and means for performing emotion analysis. This makes it possible to determine the veracity of the salesperson's words and actions in real time and provide consumers with highly reliable information. Furthermore, by analyzing emotion data, it is possible to perform a comprehensive analysis that also takes into account the consumer's emotional state and make even more accurate judgments. This makes it possible to provide consumers with an environment in which they can proceed with transactions with peace of mind.

[0793] "Camera footage" refers to visual information acquired through a camera.

[0794] "Audio data" refers to sound information acquired through a microphone.

[0795] "Heart rate data" refers to numerical information about the user's heart rate measured by a heart rate sensor.

[0796] "Emotional data" refers to data that indicates a user's emotional state, obtained from their tone of voice and facial expressions.

[0797] "Preprocessing" refers to the initial stages of processing that make data easier to analyze.

[0798] A "face recognition algorithm" refers to a computational method for automatically detecting human faces from video data.

[0799] A "speech recognition algorithm" refers to a computational method for converting speech data into text.

[0800] "Synchronizing across timeframes" refers to the process of adjusting different datasets to fit within the same timeframe.

[0801] "Emotional analysis" refers to a technology that analyzes acquired emotional data to evaluate the user's emotional state.

[0802] A "facial expression analysis model" refers to a computational model used to analyze emotions and psychological states from a person's facial expressions.

[0803] A "speech analysis model" refers to a computational model used to analyze the characteristics of speech (such as tone, pitch, and speed).

[0804] "Analysis results" refer to conclusions and evaluations obtained through the analysis of data.

[0805] "Real-time" refers to a state where the time from data acquisition to the display of analysis results is extremely short.

[0806] System Configuration

[0807] This invention provides a system that allows users to wear smart glasses to determine in real time whether a salesperson's words and actions are true or false, thereby providing consumers with reliable information. The system includes smart glasses and a server.

[0808] Device (smart glasses) side

[0809] The user wears smart glasses. The smart glasses have the following hardware and software:

[0810] Camera: Captures real-time footage of the salesperson.

[0811] Heart rate sensor: Monitors the user's heart rate.

[0812] Mike: Record the conversation.

[0813] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[0814] These devices transmit data to the server via Wi-Fi or Bluetooth.

[0815] Server side

[0816] The server receives data sent from the smart glasses and performs the following processing:

[0817] Preprocessing: Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted to text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames over time, and user emotion data obtained by the emotion engine is also preprocessed.

[0818] Data Analysis: We analyze the subtle facial movements of salespeople using facial expression analysis models (e.g., Dlib, DeepFace), and analyze the characteristics of voice data (tone, pitch, speed, etc.) using voice analysis models. We detect abnormal fluctuations (sudden increases and irregularities in heart rate) from heart rate data and analyze the user's emotional data.

[0819] Results Integration and Feedback: The analysis results are integrated to calculate the probability that the salesperson is lying. The result is calculated as a percentage, and a message such as "There is a 70% chance that they are lying" is generated. This feedback message is sent to the smart glasses in real time to notify the user of the result.

[0820] User side

[0821] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's words and actions and proceed with the contract with confidence.

[0822] Specific example

[0823] As a concrete example, consider the case of purchasing an expensive item (such as jewelry) and determining whether the salesperson's explanation is true. For instance, if a salesperson says, "This diamond is 100% pure and clean-cut," the "Trust Salesperson" application analyzes the information and displays "90% probability of being true" on the smart glasses' display. Based on this information, the customer can proceed with the purchase with confidence.

[0824] Example of a prompt

[0825] A salesperson explains, "This diamond is 100% pure and clean-cut." The customer is wearing smart glasses and using the "Trusted Salesperson" app, which analyzes the conversation, facial expressions, and voice characteristics in real time. The app indicates that there is a 99% probability that the salesperson is telling the truth. What does the customer do next?

[0826] By inputting this prompt into the AI ​​model, it is possible to support the dialogue between sales representatives and customers and facilitate reliable transactions.

[0827] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0828] Step 1:

[0829] The device (smart glasses) acquires camera footage, audio data, heart rate data, and emotion data. The camera captures real-time video of the salesperson, and the microphone records conversations. The heart rate sensor monitors the user's heart rate, and the emotion engine acquires emotion data from the user's voice tone and facial expressions. This data is transmitted to a server via Wi-Fi or Bluetooth.

[0830] Inputs: Camera video, audio data, heart rate data, emotion data

[0831] Output: Data transmission from smart glasses to server

[0832] Step 2:

[0833] The server receives data transmitted from the terminal. The received data is first pre-processed. Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames in time, and emotion data is pre-processed.

[0834] Input: Raw data sent from the terminal

[0835] Output: Preprocessed data (frame-divided video, audio converted to text, synchronized heart rate data, preprocessed emotion data)

[0836] Step 3:

[0837] The server analyzes the pre-processed data. It uses facial expression analysis models (e.g., Dlib, DeepFace) to analyze the subtle facial movements of salespeople, and voice analysis models to analyze the characteristics of voice data (tone, pitch, speed, etc.). It detects abnormal fluctuations (sudden increases or irregularities in heart rate) from heart rate data and analyzes the user's emotional data.

[0838] Input: Preprocessed data

[0839] Output: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[0840] Step 4:

[0841] The server integrates the analysis results and calculates the probability that the salesperson is lying. A statistical model is used to comprehensively evaluate parameters obtained from different analyses in the integration of the analysis results. As a result, messages such as "There is a 70% probability that they are lying" are generated.

[0842] Input: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[0843] Output: Integrated analysis results (probability of lying)

[0844] Step 5:

[0845] The server transmits the integrated analysis results to the smart glasses in real time and notifies the user of the results. The notification is displayed as a message on the smart glasses' display.

[0846] Input: Integrated analysis results

[0847] Output: Feedback message displayed on smart glasses

[0848] Step 6:

[0849] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. This allows users to verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0850] Input: Feedback message displayed on smart glasses

[0851] Output: User actions (additional questions, contract progress)

[0852] Through the steps described above, the present invention provides an environment in which consumers can quickly and accurately determine the truthfulness of a salesperson's words and actions, and proceed with transactions with confidence.

[0853] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0854] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0855] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0856] [Third Embodiment]

[0857] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0858] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0859] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0860] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0861] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0862] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0863] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0864] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0865] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0866] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0867] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0868] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0869] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. This system acquires camera video, audio data, and heart rate data, transmits them to a server for analysis, and displays the analysis results in real time to the user wearing smart glasses.

[0870] Device (smart glasses) side

[0871] First, the user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, and microphone, and these devices are activated to begin data collection. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[0872] Server side

[0873] The server receives camera video, audio data, and heart rate data sent from the smart glasses. The received data is first pre-processed. This pre-processing includes dividing the video data into frames and applying a face recognition algorithm to detect the salesperson's face. The audio data is converted into text data using a speech recognition algorithm, and the heart rate data is synchronized over time.

[0874] Next, the server analyzes the pre-processed data. Specifically, it uses a facial expression analysis model to analyze the subtle movements of the salesperson's face and extract characteristics of stress and tension. It also uses a voice analysis model to analyze the characteristics of the voice data (tone, pitch, speed, etc.) and detects abnormal fluctuations from the heart rate data. By integrating these analysis results, it calculates the probability that the salesperson is lying.

[0875] The analysis results are calculated as specific percentages, generating messages such as "There is a 70% chance you are lying." The server sends this feedback message to the smart glasses in real time, notifying the user of the results.

[0876] User side

[0877] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0878] Specific example

[0879] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[0880] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk.

[0881] The following describes the processing flow.

[0882] Device (smart glasses) side

[0883] Step 1: Data Acquisition

[0884] The smart glasses activate the camera, heart rate sensor, and microphone. The camera captures the salesperson's image at 30 frames per second, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation.

[0885] Step 2: Send to the server

[0886] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, and audio data are transferred to the server via Wi-Fi or Bluetooth.

[0887] Server side

[0888] Step 3: Data Received

[0889] The server receives data sent from the smart glasses. All camera video data, audio data, and heart rate data are received using a secure communication protocol.

[0890] Step 4: Pretreatment

[0891] The server preprocesses the received data. Specifically, it performs the following processes:

[0892] The camera footage is divided into frames, and a face recognition algorithm is applied to detect the salesperson's face.

[0893] The audio data is converted to text using a speech recognition algorithm.

[0894] The heart rate data is synchronized with the video frames over time.

[0895] Step 5: Analysis

[0896] The server analyzes the pre-processed data. The specific operation is as follows:

[0897] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople to extract indicators of stress and tension.

[0898] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[0899] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[0900] Step 6: Truth Determination

[0901] The server integrates the results of facial expression analysis, voice analysis, and heart rate analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[0902] Step 7: Generate Feedback

[0903] The server generates a feedback message to notify the user based on the truth / false determination result. This message, in text format, might say something like, "There's a 70% chance you're lying."

[0904] Step 8: Submitting the analysis results

[0905] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[0906] Device (smart glasses) side

[0907] Step 9: Display User Interface

[0908] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[0909] User side

[0910] Step 10: Checking Feedback

[0911] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[0912] Step 11: Contract Decision

[0913] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[0914] Through this series of steps, a system will be created that allows consumers to verify the veracity of the salesperson's words and actions in real time, enabling them to proceed with the contract with confidence.

[0915] (Example 1)

[0916] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0917] In sales activities, it is essential to determine the veracity of a salesperson's words and actions in real time and provide consumers with reliable information. However, currently, it is difficult to instantly judge subtle psychological states such as a salesperson's stress or dishonesty. Unless this problem is solved, there is a risk that consumers will make important decisions based on inaccurate information.

[0918] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0919] In this invention, the server includes means for dividing camera images into frames, means for applying a face recognition algorithm, means for applying a speech recognition algorithm that converts audio data into text data, and means for synchronizing heart rate data over time. This makes it possible to analyze the words and actions of salespeople in real time and to quickly and accurately determine their authenticity.

[0920] "Means of acquiring camera footage" refers to devices or methods for recording the actions of salespeople in real time during sales activities.

[0921] "Means for acquiring audio data" refers to devices and methods for recording conversations between salespeople and consumers and collecting audio information for analysis.

[0922] "Means of acquiring heart rate data" refers to devices and methods for measuring a consumer's heart rate in real time and monitoring their stress and psychological state.

[0923] "Means of sending data to a server" refers to communication technologies and protocols used to send camera video, audio data, and heart rate data to a server.

[0924] "Means for pre-processing received data" refers to processing techniques used on the server side to convert various types of data received into a format that can be analyzed.

[0925] "Means for analyzing pre-processed data" refers to methods and devices for determining the veracity of a salesperson's statements and actions by applying various analytical algorithms to pre-processed data.

[0926] "Means of displaying analysis results" refers to displays and notification systems that communicate analysis results to consumers in real time.

[0927] "Methods for dividing camera footage into frames" refers to techniques or methods for breaking down continuous video data into individual still images (frames).

[0928] "Means for applying a face recognition algorithm" refers to algorithms and methods for executing them to detect and identify faces from acquired video data.

[0929] "Means of applying speech recognition algorithms that convert audio data into text data" refers to technology that analyzes audio data and converts its content into text format.

[0930] "Methods for synchronizing heart rate data over time" refers to technologies and methods for aligning heart rate data with other data (such as video or audio) on the same time axis.

[0931] "Methods of analysis using facial expression analysis models" refer to algorithms and technologies that analyze subtle facial movements to infer emotions and psychological states.

[0932] "Methods for analyzing tone, pitch, and speed using speech analysis models" refers to algorithms and technologies for analyzing the characteristics of speech data, such as tone, pitch, and speaking speed.

[0933] "Means for analyzing heart rate data" refers to methods or devices for analyzing acquired heart rate data and determining the physiological response of consumers.

[0934] "Methods for integrating multiple analysis results to calculate the probability that a salesperson is lying" refers to algorithms and technologies that integrate the results of facial expression analysis, voice analysis, and heart rate analysis to make an overall judgment.

[0935] "Means of notifying users of analysis results in real time" refers to communication technologies and display systems that immediately convey the results of the analysis to the user.

[0936] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. The system acquires camera video, audio data, and heart rate data, and sends this data to a server for analysis. The analysis results are displayed in real time to the user wearing smart glasses.

[0937] Device (smart glasses) side

[0938] The user first puts on the smart glasses. The smart glasses include the following devices:

[0939] Camera: Captures real-time footage of the salesperson.

[0940] Heart rate sensor: Monitors the user's heart rate.

[0941] Mike: Record the conversation with the salesman.

[0942] This data is transmitted to a server using Wi-Fi or Bluetooth. For example, while a salesperson is giving a product presentation, this video, audio, and heart rate data is collected in real time.

[0943] Server side

[0944] The server receives camera video, audio data, and heart rate data sent from the smart glasses. Specifically, it uses the following hardware and software.

[0945] Preprocessing of video data: The camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of the salespeople.

[0946] Audio data preprocessing: Convert audio data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[0947] Preprocessing of heart rate data: Synchronize heart rate data with video and audio data over time.

[0948] Next, the preprocessed data is analyzed. Specific analysis includes the following:

[0949] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[0950] A speech analysis model (e.g., Praat) is used to analyze characteristics such as tone, pitch, and speed.

[0951] Heart rate data analysis: Detects abnormal fluctuations.

[0952] These analysis results are combined to calculate the probability that the salesperson is lying. The results are displayed as a percentage, such as "there is a 70% probability that they are lying."

[0953] User side

[0954] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can ask additional questions or request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[0955] Specific example

[0956] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and listens to the salesperson's explanation. During this time, the smart glasses' camera records the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes it in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use this information to ask the salesperson for clarification. This system allows users to reliably verify important contract details.

[0957] Example of a prompt

[0958] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[0959] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0960] Step 1: Data Collection

[0961] When a user puts on the smart glasses, the device simultaneously activates the camera, heart rate sensor, and microphone to begin collecting data. Inputs include the user's actions and their interaction with the salesperson. The camera captures the salesperson's video in real time, the heart rate sensor measures the user's heart rate in real time, and the microphone records the conversation. Outputs include captured video data, recorded audio data, and measured heart rate data. For example, video and audio of the moment the salesperson is explaining the product's features, along with the user's heart rate, are collected.

[0962] Step 2: Data transmission

[0963] The device transmits collected video, audio, and heart rate data to the server using Wi-Fi or Bluetooth. The input consists of the various data collected in step 1. Once the data transmission is complete, the data is received by the server as output. Specifically, the communication module of the smart glasses packets the data and sends it to the server.

[0964] Step 3: Data Preprocessing

[0965] The server receives data sent from the terminal. Inputs include camera video, audio data, and heart rate data transmitted in the data transmission step. The received data is first pre-processed. Specifically, the server divides the video data into frames and uses face recognition algorithms such as OpenCV to detect the salesperson's face. Audio data is converted to text data using the Google Speech-to-Text API, etc. Heart rate data is synchronized over time. Outputs include video frames in which faces were detected, the converted audio data, and the synchronized heart rate data.

[0966] Step 4: Data Analysis

[0967] The server analyzes the pre-processed data. The inputs include face detection video frames, text data, and synchronized heart rate data, all pre-processed in step 3. Specifically, it performs the following analyses:

[0968] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[0969] A speech analysis model (e.g., Praat) is used to analyze speech characteristics such as tone, pitch, and speed.

[0970] It detects abnormal fluctuations from heart rate data.

[0971] These analysis results are integrated to calculate the probability that the salesperson is lying. The output will include each analysis result and the integrated probability value. For example, a conclusion such as "there is a 70% probability that the salesperson is lying" might be drawn.

[0972] Step 5: Result Generation

[0973] The server generates a specific message based on the analysis results. The input is the probability value, which is an integrated version of the analysis results obtained in step 4. The generated message will display a percentage, such as "There is a 70% chance that you are lying." This message is obtained as output. Specifically, the server executes a message generation script and converts it into a user-friendly display format.

[0974] Step 6: Notification of Results

[0975] The server sends the generated analysis result message to the terminal (smart glasses) in real time. The input is the message generated in step 5. The sent message is displayed on the smart glasses' display and notified to the user. The output is the message displayed on the smart glasses. Specifically, the server's notification system packets the message and sends it to the terminal.

[0976] Step 7: User response

[0977] The user interacts with the salesperson while referring to the analysis results displayed on the smart glasses. The input is the analysis results displayed on the smart glasses. Based on the analysis results, the user can ask additional questions or request more detailed explanations. The output provides the user with information to make a more confident decision. Specifically, the user can ask the salesperson follow-up questions to confirm details.

[0978] Example of a prompt

[0979] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[0980] (Application Example 1)

[0981] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0982] Currently, consumers have limited means to determine the veracity of information provided in sales activities in real time, making it difficult to ensure consumer trust. Furthermore, directly verifying the truthfulness of salespeople's statements and actions is also difficult, posing risks when making important decisions. Therefore, a system is needed that allows consumers to conduct transactions with confidence.

[0983] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0984] In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a speech recognition algorithm, and means for synchronizing heart rate data over time. This makes it possible to integrate video analysis, audio analysis, and heart rate analysis to determine reliability.

[0985] "Means for acquiring camera images" refers to a function that uses cameras mounted on smart glasses or other devices to acquire images of a target in real time.

[0986] "Means for acquiring audio data" refers to a function that uses microphones built into smart glasses or other devices to record conversational audio in real time and acquire it as data.

[0987] "Means for acquiring heart rate data" refers to a function that uses heart rate sensors installed in smart glasses or other devices to acquire the user's heart rate data in real time.

[0988] "Means of sending data to the server" refers to the function of sending acquired camera video, audio data, and heart rate data to the server via the network.

[0989] "Means for analyzing received data" refers to algorithms and modules used to analyze data sent to a server and extract necessary information.

[0990] "Means for displaying analysis results" refers to a function for displaying the analysis results on the display of a device that has received the results (e.g., smart glasses).

[0991] "A means of notifying users of analysis results in real time" refers to a function that notifies users of the results in real time immediately after the analysis is completed.

[0992] "Means of preprocessing" refers to functions that clean and adjust data in order to improve the quality of data sent to the server.

[0993] "Means for applying a facial recognition algorithm" refers to an algorithm that recognizes a person's face based on video data and extracts specific features.

[0994] "Means for applying speech recognition algorithms" refers to algorithms that analyze speech data, convert it into text, and further analyze the characteristics of the speech.

[0995] "Methods for synchronizing heart rate data over time" refers to a function that links heart rate data with other data (such as video or audio data) in a temporal manner for analysis.

[0996] The "method for determining reliability by integrating video analysis, audio analysis, and heart rate analysis" is a function that comprehensively evaluates the results of multiple data analyses and calculates the reliability of a salesperson's words and actions.

[0997] "Means for calculating reliability evaluation results as probabilities" refers to a function that calculates, numerically, the degree of reliability based on integrated data analysis results.

[0998] "Means for displaying feedback messages on the smart glasses' display" refers to a function for displaying feedback messages based on analysis results on the smart glasses' display.

[0999] This invention is a system that supports sales activities in physical stores and allows consumers to verify the reliability of the information they are provided with. This system uses smart glasses as the main terminal, collecting data using a camera, microphone, and heart rate sensor, and analyzing the collected data on a server to determine the veracity of the salesperson's words and actions in real time.

[1000] Hardware configuration

[1001] Device (smart glasses)

[1002] The user wears smart glasses, and data is collected using the built-in camera, microphone, and heart rate sensor. The camera captures video of the salesperson in real time, the microphone records conversations, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth.

[1003] server

[1004] The server is responsible for the following roles:

[1005] 1. Data reception: Receives camera video, audio data, and heart rate data transmitted from the smart glasses.

[1006] 2. Data preprocessing: Video data is divided into frames, and a face recognition algorithm is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm, and heart rate data is synchronized over time.

[1007] 3. Data Analysis: A facial expression analysis model is used to analyze the subtle facial movements of the salesperson and extract characteristics of stress and tension. A voice analysis model is used to analyze the characteristics of the voice data (tone, pitch, speed, etc.), and abnormal fluctuations are also detected from heart rate data. These analysis results are integrated to calculate the probability that the salesperson is lying.

[1008] 4. Sending Results: The analysis results are calculated as specific percentages, and messages such as "There is a 70% chance that you are lying" are generated. The server sends this feedback message to the smart glasses in real time to notify the user of the results.

[1009] Software Configuration

[1010] The following software is used on the server side:

[1011] OpenCV: Used for preprocessing video data and face recognition.

[1012] Speech analysis module: For converting speech data into text and further analyzing the characteristics of the speech.

[1013] Heart rate analysis module: For detecting abnormal fluctuations in heart rate data.

[1014] Integrated analysis module: To integrate the analysis results of video, audio, and heart rate data to determine reliability.

[1015] Examples

[1016] As a concrete example, consider a scenario in a physical store where an expensive product is being described. The user wears smart glasses and receives an explanation from a store employee. The system allows the user to see the analysis results displayed on the smart glasses' screen in real time. Specifically, if it displays "there is an 80% probability that the explanation is reliable," the user can make a decision with confidence based on that information.

[1017] Example of a prompt

[1018] Next, we will show a concrete example of a prompt statement.

[1019] "We want to develop a system that analyzes the veracity of a salesperson's statements in real time when they provide detailed product descriptions, thereby supporting the provision of highly reliable information. This system will analyze camera footage, audio data, and heart rate data, and display the results on smart glasses. For example, we want the system to be usable in a scenario like this: when purchasing an expensive home appliance, the user wears smart glasses while receiving an explanation from a salesperson and can verify its reliability in real time. We would appreciate your advice on system design using sensors and analysis algorithms."

[1020] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1021] Step 1:

[1022] The user puts on smart glasses. The smart glasses activate the camera, microphone, and heart rate sensor and begin collecting data. Specifically, the camera captures video of the salesperson in real time, the microphone records the conversation, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth. The inputs are camera video, audio data, and heart rate data, and these are output as integrated data packets.

[1023] Step 2:

[1024] The server receives data packets transmitted from the smart glasses. The received data is first pre-processed. Specifically, the video data is divided into frames, the audio data is divided into fixed time units, and the heart rate data is compared with a timestamp. The input is raw data of camera video, audio data, and heart rate data, and the output is a pre-processed dataset.

[1025] Step 3:

[1026] The server applies a face recognition algorithm to pre-processed video data to detect the faces of salespeople. Specifically, it uses OpenCV to extract the position and major feature points of faces for each frame. The input is pre-processed video data, and the output is video data with face detection information added.

[1027] Step 4:

[1028] The server applies a speech recognition algorithm to the audio data, converting it into text data while simultaneously analyzing the characteristics of the speech (tone, pitch, speed, etc.). Specifically, it uses a speech recognition engine with a generative AI model to generate text from the audio data and extract speech features at the same time. The input is pre-processed audio data, and the output consists of text data and speech characteristic data.

[1029] Step 5:

[1030] The server analyzes heart rate data. This includes monitoring heart rate variability in real time and detecting any abnormalities. Specifically, it uses a heart rate analysis algorithm to evaluate heart rate data and detect signs of stress or tension. The input is pre-processed heart rate data, and the output is the analyzed heart rate data.

[1031] Step 6:

[1032] The server integrates the results of video analysis, audio analysis, and heart rate analysis to determine the reliability of the salesperson's words and actions. Specifically, it integrates the analysis results of each data to quantify the probability that the salesperson is lying. The inputs are video data with facial detection information, text data, audio characteristic data, and analyzed heart rate data, and the reliability evaluation results are output.

[1033] Step 7:

[1034] The server sends the evaluation results to the smart glasses and notifies the user. Specifically, it generates a reliability evaluation result as a feedback message and displays it on the smart glasses in real time. The input is the reliability evaluation result, and the feedback message is output. Based on the analysis results displayed on the smart glasses' screen, the user can proceed with the conversation with the salesperson. If a message such as "There is a 70% chance they are lying" is displayed, the user can ask further questions and request a more detailed explanation.

[1035] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1036] This invention is a system that uses a user emotion engine to determine the veracity of a salesperson's words and actions in real time, providing consumers with highly reliable information. This allows consumers to proceed with transactions with confidence.

[1037] Device (smart glasses) side

[1038] The user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, microphone, and emotion engine, and these devices are activated to begin collecting data. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, the microphone records the conversation, and the emotion engine obtains emotional data from the user's voice tone and facial expressions. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[1039] Server side

[1040] The server receives camera video, audio data, heart rate data, and emotion data sent from the smart glasses. The received data is first pre-processed, which includes the following:

[1041] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the faces of the salespeople.

[1042] The audio data is converted into text data using a speech recognition algorithm.

[1043] Heart rate data is synchronized with video frames over time.

[1044] User sentiment data acquired by the sentiment engine is also preprocessed.

[1045] Next, the server analyzes the pre-processed data. The specific steps are as follows:

[1046] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople and extract indicators of stress and tension.

[1047] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[1048] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[1049] We analyze user emotion data and evaluate the user's emotional state.

[1050] These analysis results are integrated to calculate the probability that the salesperson is lying. The analysis results are calculated as a specific percentage, and a message such as "There is a 70% chance that they are lying" is generated. The server sends this feedback message to the smart glasses in real time to notify the user of the result.

[1051] User side

[1052] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[1053] Specific example

[1054] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[1055] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk. Furthermore, by combining it with an emotion engine, a comprehensive analysis that also takes into account the user's emotional state can be performed, enabling even more accurate determination of truthfulness.

[1056] The following describes the processing flow.

[1057] Device (smart glasses) side

[1058] Step 1: Data Acquisition

[1059] The smart glasses activate a camera, heart rate sensor, microphone, and emotion engine. The camera captures video of the salesperson at 30 frames per second, the heart rate sensor monitors the user's heart rate, the microphone records conversations, and the emotion engine extracts emotional data from the user's voice tone and facial expressions.

[1060] Step 2: Send to the server

[1061] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, audio data, and emotion data are transferred to the server via Wi-Fi or Bluetooth.

[1062] Server side

[1063] Step 3: Data Received

[1064] The server receives data sent from the smart glasses. All data, including camera video, audio, heart rate, and emotion data, is received using a secure communication protocol.

[1065] Step 4: Pretreatment

[1066] The server preprocesses the received data. The specific steps are as follows:

[1067] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the salesperson's face.

[1068] The audio data is converted to text using a speech recognition algorithm.

[1069] The heart rate data is synchronized with the video frame and timeline.

[1070] The emotional data is formatted appropriately to make it easier to analyze.

[1071] Step 5: Facial Expression Analysis

[1072] The server inputs pre-processed video data into a facial expression analysis model to analyze the subtle movements of the salesperson's face. It then extracts features that serve as indicators of stress and tension.

[1073] Step 6: Voice Analysis

[1074] The server inputs audio data into a speech analysis model, which analyzes speech characteristics such as tone, pitch, and speed. It also detects signs of stress and tension.

[1075] Step 7: Heart Rate Analysis

[1076] The server analyzes heart rate data and detects abnormal fluctuations such as sudden increases or irregularities in the user's heart rate.

[1077] Step 8: Emotion Analysis

[1078] The server analyzes pre-processed emotional data to evaluate the user's emotional state. It identifies emotional responses from the user's tone of voice and facial expressions.

[1079] Step 9: Comprehensive analysis and truth / false determination

[1080] The server integrates the results of facial expression analysis, voice analysis, heart rate analysis, and emotion analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[1081] Step 10: Generate Feedback

[1082] Based on the truth / false determination result, the server generates a feedback message to notify the user. This message, in text format, might state something like, "There is a 70% chance you are lying."

[1083] Step 11: Submitting the analysis results

[1084] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[1085] Device (smart glasses) side

[1086] Step 12: Display User Interface

[1087] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[1088] User side

[1089] Step 13: Checking Feedback

[1090] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[1091] Step 14: Decision on the contract

[1092] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[1093] Through the above series of steps, a system will be realized that allows consumers to verify the veracity of salespeople's words and actions in real time, enabling them to proceed with contracts with confidence. Furthermore, the introduction of an emotion engine will enable comprehensive analysis that also takes into account the user's emotional state, resulting in even more accurate determination of truthfulness.

[1094] (Example 2)

[1095] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1096] In traditional communication between consumers and salespeople, it was difficult to accurately determine the truthfulness of the salesperson's words and actions, resulting in a high risk of consumers suffering disadvantages. This problem prevented consumers from proceeding with transactions with confidence, especially in financial transactions and high-value contracts. Furthermore, traditional systems struggled to perform comprehensive analysis that took into account the user's emotional state, resulting in low accuracy in determining truthfulness.

[1097] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1098] In this invention, the server includes means for acquiring the user's biometric data, such as heart rate data; means for acquiring the user's emotional state; means for transmitting the data to the server via a communication network; means for pre-processing the received data; and means for integrating the analysis results of the pre-processed data and determining its truthfulness. This makes it possible to quickly and accurately determine the truthfulness of a salesperson's words and actions, allowing consumers to proceed with transactions with peace of mind. Furthermore, by performing a comprehensive analysis that also takes the user's emotional state into consideration, it becomes possible to determine truthfulness with even greater accuracy.

[1099] "Camera footage" refers to video data acquired by a camera installed in a wearable device worn by the user.

[1100] "Voice data" refers to audio information acquired by a microphone installed in a wearable device worn by the user.

[1101] "Heart rate data" refers to data obtained by measuring the user's heart rate, which is a biometric piece of information.

[1102] "Emotional state" refers to the psychological state of a user, obtained by analyzing their voice tone, facial expressions, and other biometric information.

[1103] A "communication network" refers to the infrastructure used to send and receive data, and this includes Wi-Fi, Bluetooth, and other similar technologies.

[1104] "Preprocessing" refers to a series of processes used to convert acquired raw data into an analyzable format, and includes data splitting, synchronization, and normalization.

[1105] "Analysis" is the process of evaluating the characteristics and relationships of pre-processed data using specific algorithms and models.

[1106] "Determining truthfulness" involves calculating the probability of whether a salesperson's statements or actions are true or false based on the analyzed data.

[1107] "Feedback" refers to a means of providing information to notify users of analysis results, and this includes visual displays and audio notifications.

[1108] "Integration" refers to the process of combining different types of pre-processed and analyzed data (such as video, audio, heart rate, and emotions) into a single, comprehensive analysis.

[1109] This invention is a system designed to allow users to proceed with transactions with confidence during interactions with sales representatives. This system uses a wearable device worn by the user (e.g., smart glasses) and a server to determine the veracity of the sales representative's statements in real time. The specific configuration and operation of the entire system are described below.

[1110] Hardware configuration

[1111] Device side (smart glasses)

[1112] Camera: Captures video in real time.

[1113] Mike: Record the conversation.

[1114] Heart rate sensor: Monitors the user's heart rate.

[1115] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[1116] Communication module: Has the function of sending data to a server via Wi-Fi or Bluetooth.

[1117] Server side

[1118] Data receiving module: Receives data transmitted from smart glasses.

[1119] Preprocessing module: Preprocesses the received data.

[1120] Analysis module: Analyzes pre-processed data.

[1121] Feedback generation module: Generates feedback based on the analysis results.

[1122] Communication module: Sends the generated feedback to the smart glasses.

[1123] Data processing and data calculation

[1124] 1. The user puts on the smart glasses and begins the conversation.

[1125] 2. The device (smart glasses) collects camera video, audio data, heart rate data, and emotion data, and transmits them to the server in real time.

[1126] 3. The server divides the received video data into frames and applies a face recognition algorithm to detect the salesperson's face. For example, it uses computer vision technology such as OpenFace.

[1127] 4. The server converts the audio data into text data using a speech recognition algorithm. For example, it uses deep learning-based speech recognition technology such as Kaldi.

[1128] 5. The server uses techniques such as the Short-Term Fourier Transform to synchronize heart rate data and video frames over time.

[1129] 6. The server normalizes the emotional data and evaluates the user's psychological state as numerical data. It uses an emotional analysis model such as EmotionRecognitionNet.

[1130] 7. The server analyzes the pre-processed data and evaluates the characteristics of each data point using facial expression analysis models, voice analysis models, and heart rate analysis models.

[1131] 8. The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[1132] 9. The server generates specific feedback messages based on probability, such as "There is a 70% chance you are lying," and sends these to the smart glasses.

[1133] 10. The user uses the feedback displayed on the smart glasses to guide the conversation. For example, they might ask the salesperson additional questions based on the feedback to verify the accuracy of the information.

[1134] Specific example

[1135] As a concrete example, consider the case of a high-value mortgage contract. When a user wears smart glasses and receives an explanation from a salesperson, the smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server in real time and analyzed immediately. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use the warning message displayed on the smart glasses to ask the salesperson for clarification again.

[1136] Example of a prompt

[1137] Please provide a detailed explanation of what a user should do if, during a high-value mortgage contract, smart glasses detect an 80% probability that the salesperson is lying while explaining the benefits.

[1138] The above describes a specific embodiment of the present invention. This system allows users to quickly and accurately determine the truthfulness of a salesperson's words and actions, enabling them to proceed with transactions with confidence. Furthermore, since the analysis also takes into account the user's emotional state, even more accurate determinations are possible.

[1139] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1140] Program processing flow

[1141] Step 1: Device startup and data collection

[1142] input:

[1143] The user puts on smart glasses.

[1144] process:

[1145] The camera on the device (smart glasses) activates and begins recording video in real time.

[1146] The microphone activates and begins recording the conversation.

[1147] The heart rate sensor begins measuring the user's heart rate.

[1148] The emotion engine begins analyzing the user's voice tone and facial expressions in real time.

[1149] output:

[1150] Collected video data, audio data, heart rate data, and emotion data.

[1151] Step 2: Send

[1152] input:

[1153] Collected video data, audio data, heart rate data, and emotion data.

[1154] process:

[1155] The device sends data to the server via Wi-Fi or Bluetooth.

[1156] The communication module compresses the data and sends it to the server via a secure network.

[1157] output:

[1158] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[1159] Step 3: Data Preprocessing

[1160] input:

[1161] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[1162] process:

[1163] The server divides the video data into frames and applies a face recognition algorithm (e.g., OpenFace) to detect the salesperson's face.

[1164] The server converts the audio data into text data using a speech recognition algorithm (e.g., Kaldi).

[1165] The server applies a short-term Fourier transform to synchronize the heart rate data with the video frame.

[1166] Emotional data is normalized, and the user's psychological state is evaluated as numerical data.

[1167] output:

[1168] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[1169] Step 4: Data Analysis

[1170] input:

[1171] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[1172] process:

[1173] The server uses a facial expression analysis model to analyze the subtle facial movements of the salesperson and extract indicators of stress and tension.

[1174] The server uses a speech analysis model to analyze the characteristics of the speech (tone, pitch, speed, etc.).

[1175] The server detects abnormal fluctuations from the heart rate data.

[1176] The server analyzes emotional data and evaluates the user's emotional state.

[1177] output:

[1178] Salesperson stress and tension indicators, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[1179] Step 5: Truth Determination and Feedback Generation

[1180] input:

[1181] Analyzed stress and tension indices, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[1182] process:

[1183] The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[1184] The server generates feedback messages expressing the truthfulness determination result as a specific percentage, such as "There is a 70% chance that they are lying."

[1185] output:

[1186] Feedback message (e.g., "There's a 70% chance you're lying").

[1187] Step 6: Displaying Feedback and User Interaction

[1188] input:

[1189] Feedback message.

[1190] process:

[1191] The device displays a feedback message on the smart glasses' display.

[1192] Voice notifications will also be provided as needed.

[1193] output:

[1194] A feedback message displayed to the user.

[1195] Step 7: User response

[1196] input:

[1197] The displayed feedback message.

[1198] process:

[1199] Users use the feedback messages to ask salespeople additional questions.

[1200] Users will use feedback messages to proceed with the dialogue more carefully.

[1201] output:

[1202] Asking additional questions to the salesperson and facilitating the conversation.

[1203] Through the above processing steps, the system can determine the veracity of the salesperson's statements in real time and provide feedback to the user. This allows the user to proceed with transactions with confidence.

[1204] (Application Example 2)

[1205] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1206] In sales activities, consumers often find it difficult to quickly and accurately determine the truthfulness of a salesperson's words and actions, making it difficult to evaluate the credibility of sales pitches and explanations, which can lead to disadvantages. Furthermore, it is not easy for consumers to understand their own emotional state and make calm judgments. In addition, there is a lack of tools to determine whether what a salesperson is saying is trustworthy, making it difficult to create an environment where consumers can proceed with transactions with peace of mind.

[1207] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a voice recognition algorithm, means for synchronizing heart rate data over time, and means for performing emotion analysis. This makes it possible to determine the veracity of the salesperson's words and actions in real time and provide consumers with highly reliable information. Furthermore, by analyzing emotion data, it is possible to perform a comprehensive analysis that also takes into account the consumer's emotional state and make even more accurate judgments. This makes it possible to provide consumers with an environment in which they can proceed with transactions with peace of mind.

[1208] "Camera footage" refers to visual information acquired through a camera.

[1209] "Audio data" refers to sound information acquired through a microphone.

[1210] "Heart rate data" refers to numerical information about the user's heart rate measured by a heart rate sensor.

[1211] "Emotional data" refers to data that indicates a user's emotional state, obtained from their tone of voice and facial expressions.

[1212] "Preprocessing" refers to the initial stages of processing that make data easier to analyze.

[1213] A "face recognition algorithm" refers to a computational method for automatically detecting human faces from video data.

[1214] A "speech recognition algorithm" refers to a computational method for converting speech data into text.

[1215] "Synchronizing across timeframes" refers to the process of adjusting different datasets to fit within the same timeframe.

[1216] "Emotional analysis" refers to a technology that analyzes acquired emotional data to evaluate the user's emotional state.

[1217] A "facial expression analysis model" refers to a computational model used to analyze emotions and psychological states from a person's facial expressions.

[1218] A "speech analysis model" refers to a computational model used to analyze the characteristics of speech (such as tone, pitch, and speed).

[1219] "Analysis results" refer to conclusions and evaluations obtained through the analysis of data.

[1220] "Real-time" refers to a state where the time from data acquisition to the display of analysis results is extremely short.

[1221] System Configuration

[1222] This invention provides a system that allows users to wear smart glasses to determine in real time whether a salesperson's words and actions are true or false, thereby providing consumers with reliable information. The system includes smart glasses and a server.

[1223] Device (smart glasses) side

[1224] The user wears smart glasses. The smart glasses have the following hardware and software:

[1225] Camera: Captures real-time footage of the salesperson.

[1226] Heart rate sensor: Monitors the user's heart rate.

[1227] Mike: Record the conversation.

[1228] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[1229] These devices transmit data to the server via Wi-Fi or Bluetooth.

[1230] Server side

[1231] The server receives data sent from the smart glasses and performs the following processing:

[1232] Preprocessing: Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted to text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames over time, and user emotion data obtained by the emotion engine is also preprocessed.

[1233] Data Analysis: We analyze the subtle facial movements of salespeople using facial expression analysis models (e.g., Dlib, DeepFace), and analyze the characteristics of voice data (tone, pitch, speed, etc.) using voice analysis models. We detect abnormal fluctuations (sudden increases and irregularities in heart rate) from heart rate data and analyze the user's emotional data.

[1234] Results Integration and Feedback: The analysis results are integrated to calculate the probability that the salesperson is lying. The result is calculated as a percentage, and a message such as "There is a 70% chance that they are lying" is generated. This feedback message is sent to the smart glasses in real time to notify the user of the result.

[1235] User side

[1236] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's words and actions and proceed with the contract with confidence.

[1237] Specific example

[1238] As a concrete example, consider the case of purchasing an expensive item (such as jewelry) and determining whether the salesperson's explanation is true. For instance, if a salesperson says, "This diamond is 100% pure and clean-cut," the "Trust Salesperson" application analyzes the information and displays "90% probability of being true" on the smart glasses' display. Based on this information, the customer can proceed with the purchase with confidence.

[1239] Example of a prompt

[1240] A salesperson explains, "This diamond is 100% pure and clean-cut." The customer is wearing smart glasses and using the "Trusted Salesperson" app, which analyzes the conversation, facial expressions, and voice characteristics in real time. The app indicates that there is a 99% probability that the salesperson is telling the truth. What does the customer do next?

[1241] By inputting this prompt into the AI ​​model, it is possible to support the dialogue between sales representatives and customers and facilitate reliable transactions.

[1242] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1243] Step 1:

[1244] The device (smart glasses) acquires camera footage, audio data, heart rate data, and emotion data. The camera captures real-time video of the salesperson, and the microphone records conversations. The heart rate sensor monitors the user's heart rate, and the emotion engine acquires emotion data from the user's voice tone and facial expressions. This data is transmitted to a server via Wi-Fi or Bluetooth.

[1245] Inputs: Camera video, audio data, heart rate data, emotion data

[1246] Output: Data transmission from smart glasses to server

[1247] Step 2:

[1248] The server receives data transmitted from the terminal. The received data is first pre-processed. Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames in time, and emotion data is pre-processed.

[1249] Input: Raw data sent from the terminal

[1250] Output: Preprocessed data (frame-divided video, audio converted to text, synchronized heart rate data, preprocessed emotion data)

[1251] Step 3:

[1252] The server analyzes the pre-processed data. It uses facial expression analysis models (e.g., Dlib, DeepFace) to analyze the subtle facial movements of salespeople, and voice analysis models to analyze the characteristics of voice data (tone, pitch, speed, etc.). It detects abnormal fluctuations (sudden increases or irregularities in heart rate) from heart rate data and analyzes the user's emotional data.

[1253] Input: Preprocessed data

[1254] Output: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[1255] Step 4:

[1256] The server integrates the analysis results and calculates the probability that the salesperson is lying. A statistical model is used to comprehensively evaluate parameters obtained from different analyses in the integration of the analysis results. As a result, messages such as "There is a 70% probability that they are lying" are generated.

[1257] Input: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[1258] Output: Integrated analysis results (probability of lying)

[1259] Step 5:

[1260] The server transmits the integrated analysis results to the smart glasses in real time and notifies the user of the results. The notification is displayed as a message on the smart glasses' display.

[1261] Input: Integrated analysis results

[1262] Output: Feedback message displayed on smart glasses

[1263] Step 6:

[1264] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. This allows users to verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[1265] Input: Feedback message displayed on smart glasses

[1266] Output: User actions (additional questions, contract progress)

[1267] Through the steps described above, the present invention provides an environment in which consumers can quickly and accurately determine the truthfulness of a salesperson's words and actions, and proceed with transactions with confidence.

[1268] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1269] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1270] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1271] [Fourth Embodiment]

[1272] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1273] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1274] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1275] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1276] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1277] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1278] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1279] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1280] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1281] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1282] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1283] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1284] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1285] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. This system acquires camera video, audio data, and heart rate data, transmits them to a server for analysis, and displays the analysis results in real time to the user wearing smart glasses.

[1286] Device (smart glasses) side

[1287] First, the user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, and microphone, and these devices are activated to begin data collection. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[1288] Server side

[1289] The server receives camera video, audio data, and heart rate data sent from the smart glasses. The received data is first pre-processed. This pre-processing includes dividing the video data into frames and applying a face recognition algorithm to detect the salesperson's face. The audio data is converted into text data using a speech recognition algorithm, and the heart rate data is synchronized over time.

[1290] Next, the server analyzes the pre-processed data. Specifically, it uses a facial expression analysis model to analyze the subtle movements of the salesperson's face and extract characteristics of stress and tension. It also uses a voice analysis model to analyze the characteristics of the voice data (tone, pitch, speed, etc.) and detects abnormal fluctuations from the heart rate data. By integrating these analysis results, it calculates the probability that the salesperson is lying.

[1291] The analysis results are calculated as specific percentages, generating messages such as "There is a 70% chance you are lying." The server sends this feedback message to the smart glasses in real time, notifying the user of the results.

[1292] User side

[1293] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[1294] Specific example

[1295] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[1296] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk.

[1297] The following describes the processing flow.

[1298] Device (smart glasses) side

[1299] Step 1: Data Acquisition

[1300] The smart glasses activate the camera, heart rate sensor, and microphone. The camera captures the salesperson's image at 30 frames per second, the heart rate sensor monitors the user's heart rate, and the microphone records the conversation.

[1301] Step 2: Send to the server

[1302] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, and audio data are transferred to the server via Wi-Fi or Bluetooth.

[1303] Server side

[1304] Step 3: Data Received

[1305] The server receives data sent from the smart glasses. All camera video data, audio data, and heart rate data are received using a secure communication protocol.

[1306] Step 4: Pretreatment

[1307] The server preprocesses the received data. Specifically, it performs the following processes:

[1308] The camera footage is divided into frames, and a face recognition algorithm is applied to detect the salesperson's face.

[1309] The audio data is converted to text using a speech recognition algorithm.

[1310] The heart rate data is synchronized with the video frames over time.

[1311] Step 5: Analysis

[1312] The server analyzes the pre-processed data. The specific operation is as follows:

[1313] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople to extract indicators of stress and tension.

[1314] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[1315] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[1316] Step 6: Truth Determination

[1317] The server integrates the results of facial expression analysis, voice analysis, and heart rate analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[1318] Step 7: Generate Feedback

[1319] The server generates a feedback message to notify the user based on the truth / false determination result. This message, in text format, might say something like, "There's a 70% chance you're lying."

[1320] Step 8: Submitting the analysis results

[1321] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[1322] Device (smart glasses) side

[1323] Step 9: Display User Interface

[1324] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[1325] User side

[1326] Step 10: Checking Feedback

[1327] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[1328] Step 11: Contract Decision

[1329] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[1330] Through this series of steps, a system will be created that allows consumers to verify the veracity of the salesperson's words and actions in real time, enabling them to proceed with the contract with confidence.

[1331] (Example 1)

[1332] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1333] In sales activities, it is essential to determine the veracity of a salesperson's words and actions in real time and provide consumers with reliable information. However, currently, it is difficult to instantly judge subtle psychological states such as a salesperson's stress or dishonesty. Unless this problem is solved, there is a risk that consumers will make important decisions based on inaccurate information.

[1334] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1335] In this invention, the server includes means for dividing camera images into frames, means for applying a face recognition algorithm, means for applying a speech recognition algorithm that converts audio data into text data, and means for synchronizing heart rate data over time. This makes it possible to analyze the words and actions of salespeople in real time and to quickly and accurately determine their authenticity.

[1336] "Means of acquiring camera footage" refers to devices or methods for recording the actions of salespeople in real time during sales activities.

[1337] "Means for acquiring audio data" refers to devices and methods for recording conversations between salespeople and consumers and collecting audio information for analysis.

[1338] "Means of acquiring heart rate data" refers to devices and methods for measuring a consumer's heart rate in real time and monitoring their stress and psychological state.

[1339] "Means of sending data to a server" refers to communication technologies and protocols used to send camera video, audio data, and heart rate data to a server.

[1340] "Means for pre-processing received data" refers to processing techniques used on the server side to convert various types of data received into a format that can be analyzed.

[1341] "Means for analyzing pre-processed data" refers to methods and devices for determining the veracity of a salesperson's statements and actions by applying various analytical algorithms to pre-processed data.

[1342] "Means of displaying analysis results" refers to displays and notification systems that communicate analysis results to consumers in real time.

[1343] "Methods for dividing camera footage into frames" refers to techniques or methods for breaking down continuous video data into individual still images (frames).

[1344] "Means for applying a face recognition algorithm" refers to algorithms and methods for executing them to detect and identify faces from acquired video data.

[1345] "Means of applying speech recognition algorithms that convert audio data into text data" refers to technology that analyzes audio data and converts its content into text format.

[1346] "Methods for synchronizing heart rate data over time" refers to technologies and methods for aligning heart rate data with other data (such as video or audio) on the same time axis.

[1347] "Methods of analysis using facial expression analysis models" refer to algorithms and technologies that analyze subtle facial movements to infer emotions and psychological states.

[1348] "Methods for analyzing tone, pitch, and speed using speech analysis models" refers to algorithms and technologies for analyzing the characteristics of speech data, such as tone, pitch, and speaking speed.

[1349] "Means for analyzing heart rate data" refers to methods or devices for analyzing acquired heart rate data and determining the physiological response of consumers.

[1350] "Methods for integrating multiple analysis results to calculate the probability that a salesperson is lying" refers to algorithms and technologies that integrate the results of facial expression analysis, voice analysis, and heart rate analysis to make an overall judgment.

[1351] "Means of notifying users of analysis results in real time" refers to communication technologies and display systems that immediately convey the results of the analysis to the user.

[1352] This invention is a system that determines the veracity of a salesperson's words and actions in real time and provides consumers with reliable information. The system acquires camera video, audio data, and heart rate data, and sends this data to a server for analysis. The analysis results are displayed in real time to the user wearing smart glasses.

[1353] Device (smart glasses) side

[1354] The user first puts on the smart glasses. The smart glasses include the following devices:

[1355] Camera: Captures real-time footage of the salesperson.

[1356] Heart rate sensor: Monitors the user's heart rate.

[1357] Mike: Record the conversation with the salesman.

[1358] This data is transmitted to a server using Wi-Fi or Bluetooth. For example, while a salesperson is giving a product presentation, this video, audio, and heart rate data is collected in real time.

[1359] Server side

[1360] The server receives camera video, audio data, and heart rate data sent from the smart glasses. Specifically, it uses the following hardware and software.

[1361] Preprocessing of video data: The camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of the salespeople.

[1362] Audio data preprocessing: Convert audio data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[1363] Preprocessing of heart rate data: Synchronize heart rate data with video and audio data over time.

[1364] Next, the preprocessed data is analyzed. Specific analysis includes the following:

[1365] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[1366] A speech analysis model (e.g., Praat) is used to analyze characteristics such as tone, pitch, and speed.

[1367] Heart rate data analysis: Detects abnormal fluctuations.

[1368] These analysis results are combined to calculate the probability that the salesperson is lying. The results are displayed as a percentage, such as "there is a 70% probability that they are lying."

[1369] User side

[1370] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can ask additional questions or request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[1371] Specific example

[1372] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and listens to the salesperson's explanation. During this time, the smart glasses' camera records the salesperson's image, a heart rate sensor monitors the user's heart rate, and a microphone records the conversation. This data is sent to a server, which analyzes it in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use this information to ask the salesperson for clarification. This system allows users to reliably verify important contract details.

[1373] Example of a prompt

[1374] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[1375] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1376] Step 1: Data Collection

[1377] When a user puts on the smart glasses, the device simultaneously activates the camera, heart rate sensor, and microphone to begin collecting data. Inputs include the user's actions and their interaction with the salesperson. The camera captures the salesperson's video in real time, the heart rate sensor measures the user's heart rate in real time, and the microphone records the conversation. Outputs include captured video data, recorded audio data, and measured heart rate data. For example, video and audio of the moment the salesperson is explaining the product's features, along with the user's heart rate, are collected.

[1378] Step 2: Data transmission

[1379] The device transmits collected video, audio, and heart rate data to the server using Wi-Fi or Bluetooth. The input consists of the various data collected in step 1. Once the data transmission is complete, the data is received by the server as output. Specifically, the communication module of the smart glasses packets the data and sends it to the server.

[1380] Step 3: Data Preprocessing

[1381] The server receives data sent from the terminal. Inputs include camera video, audio data, and heart rate data transmitted in the data transmission step. The received data is first pre-processed. Specifically, the server divides the video data into frames and uses face recognition algorithms such as OpenCV to detect the salesperson's face. Audio data is converted to text data using the Google Speech-to-Text API, etc. Heart rate data is synchronized over time. Outputs include video frames in which faces were detected, the converted audio data, and the synchronized heart rate data.

[1382] Step 4: Data Analysis

[1383] The server analyzes the pre-processed data. The inputs include face detection video frames, text data, and synchronized heart rate data, all pre-processed in step 3. Specifically, it performs the following analyses:

[1384] Using a facial expression analysis model (e.g., Facial Action Coding System (FACS)), we analyze the subtle facial movements of salespeople and extract characteristics of stress and tension.

[1385] A speech analysis model (e.g., Praat) is used to analyze speech characteristics such as tone, pitch, and speed.

[1386] It detects abnormal fluctuations from heart rate data.

[1387] These analysis results are integrated to calculate the probability that the salesperson is lying. The output will include each analysis result and the integrated probability value. For example, a conclusion such as "there is a 70% probability that the salesperson is lying" might be drawn.

[1388] Step 5: Result Generation

[1389] The server generates a specific message based on the analysis results. The input is the probability value, which is an integrated version of the analysis results obtained in step 4. The generated message will display a percentage, such as "There is a 70% chance that you are lying." This message is obtained as output. Specifically, the server executes a message generation script and converts it into a user-friendly display format.

[1390] Step 6: Notification of Results

[1391] The server sends the generated analysis result message to the terminal (smart glasses) in real time. The input is the message generated in step 5. The sent message is displayed on the smart glasses' display and notified to the user. The output is the message displayed on the smart glasses. Specifically, the server's notification system packets the message and sends it to the terminal.

[1392] Step 7: User response

[1393] The user interacts with the salesperson while referring to the analysis results displayed on the smart glasses. The input is the analysis results displayed on the smart glasses. Based on the analysis results, the user can ask additional questions or request more detailed explanations. The output provides the user with information to make a more confident decision. Specifically, the user can ask the salesperson follow-up questions to confirm details.

[1394] Example of a prompt

[1395] "Please describe the detailed processing steps and specific operations of a system that determines in real time the veracity of a salesperson's statements when receiving an explanation of a high-value mortgage contract."

[1396] (Application Example 1)

[1397] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1398] Currently, consumers have limited means to determine the veracity of information provided in sales activities in real time, making it difficult to ensure consumer trust. Furthermore, directly verifying the truthfulness of salespeople's statements and actions is also difficult, posing risks when making important decisions. Therefore, a system is needed that allows consumers to conduct transactions with confidence.

[1399] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1400] In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a speech recognition algorithm, and means for synchronizing heart rate data over time. This makes it possible to integrate video analysis, audio analysis, and heart rate analysis to determine reliability.

[1401] "Means for acquiring camera images" refers to a function that uses cameras mounted on smart glasses or other devices to acquire images of a target in real time.

[1402] "Means for acquiring audio data" refers to a function that uses microphones built into smart glasses or other devices to record conversational audio in real time and acquire it as data.

[1403] "Means for acquiring heart rate data" refers to a function that uses heart rate sensors installed in smart glasses or other devices to acquire the user's heart rate data in real time.

[1404] "Means of sending data to the server" refers to the function of sending acquired camera video, audio data, and heart rate data to the server via the network.

[1405] "Means for analyzing received data" refers to algorithms and modules used to analyze data sent to a server and extract necessary information.

[1406] "Means for displaying analysis results" refers to a function for displaying the analysis results on the display of a device that has received the results (e.g., smart glasses).

[1407] "A means of notifying users of analysis results in real time" refers to a function that notifies users of the results in real time immediately after the analysis is completed.

[1408] "Means of preprocessing" refers to functions that clean and adjust data in order to improve the quality of data sent to the server.

[1409] "Means for applying a facial recognition algorithm" refers to an algorithm that recognizes a person's face based on video data and extracts specific features.

[1410] "Means for applying speech recognition algorithms" refers to algorithms that analyze speech data, convert it into text, and further analyze the characteristics of the speech.

[1411] "Methods for synchronizing heart rate data over time" refers to a function that links heart rate data with other data (such as video or audio data) in a temporal manner for analysis.

[1412] The "method for determining reliability by integrating video analysis, audio analysis, and heart rate analysis" is a function that comprehensively evaluates the results of multiple data analyses and calculates the reliability of a salesperson's words and actions.

[1413] "Means for calculating reliability evaluation results as probabilities" refers to a function that calculates, numerically, the degree of reliability based on integrated data analysis results.

[1414] "Means for displaying feedback messages on the smart glasses' display" refers to a function for displaying feedback messages based on analysis results on the smart glasses' display.

[1415] This invention is a system that supports sales activities in physical stores and allows consumers to verify the reliability of the information they are provided with. This system uses smart glasses as the main terminal, collecting data using a camera, microphone, and heart rate sensor, and analyzing the collected data on a server to determine the veracity of the salesperson's words and actions in real time.

[1416] Hardware configuration

[1417] Device (smart glasses)

[1418] The user wears smart glasses, and data is collected using the built-in camera, microphone, and heart rate sensor. The camera captures video of the salesperson in real time, the microphone records conversations, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth.

[1419] server

[1420] The server is responsible for the following roles:

[1421] 1. Data reception: Receives camera video, audio data, and heart rate data transmitted from the smart glasses.

[1422] 2. Data preprocessing: Video data is divided into frames, and a face recognition algorithm is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm, and heart rate data is synchronized over time.

[1423] 3. Data Analysis: A facial expression analysis model is used to analyze the subtle facial movements of the salesperson and extract characteristics of stress and tension. A voice analysis model is used to analyze the characteristics of the voice data (tone, pitch, speed, etc.), and abnormal fluctuations are also detected from heart rate data. These analysis results are integrated to calculate the probability that the salesperson is lying.

[1424] 4. Sending Results: The analysis results are calculated as specific percentages, and messages such as "There is a 70% chance that you are lying" are generated. The server sends this feedback message to the smart glasses in real time to notify the user of the results.

[1425] Software Configuration

[1426] The following software is used on the server side:

[1427] OpenCV: Used for preprocessing video data and face recognition.

[1428] Speech analysis module: For converting speech data into text and further analyzing the characteristics of the speech.

[1429] Heart rate analysis module: For detecting abnormal fluctuations in heart rate data.

[1430] Integrated analysis module: To integrate the analysis results of video, audio, and heart rate data to determine reliability.

[1431] Examples

[1432] As a concrete example, consider a scenario in a physical store where an expensive product is being described. The user wears smart glasses and receives an explanation from a store employee. The system allows the user to see the analysis results displayed on the smart glasses' screen in real time. Specifically, if it displays "there is an 80% probability that the explanation is reliable," the user can make a decision with confidence based on that information.

[1433] Example of a prompt

[1434] Next, we will show a concrete example of a prompt statement.

[1435] "We want to develop a system that analyzes the veracity of a salesperson's statements in real time when they provide detailed product descriptions, thereby supporting the provision of highly reliable information. This system will analyze camera footage, audio data, and heart rate data, and display the results on smart glasses. For example, we want the system to be usable in a scenario like this: when purchasing an expensive home appliance, the user wears smart glasses while receiving an explanation from a salesperson and can verify its reliability in real time. We would appreciate your advice on system design using sensors and analysis algorithms."

[1436] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1437] Step 1:

[1438] The user puts on smart glasses. The smart glasses activate the camera, microphone, and heart rate sensor and begin collecting data. Specifically, the camera captures video of the salesperson in real time, the microphone records the conversation, and the heart rate sensor monitors the user's heart rate. This data is transmitted to a server via Wi-Fi or Bluetooth. The inputs are camera video, audio data, and heart rate data, and these are output as integrated data packets.

[1439] Step 2:

[1440] The server receives data packets transmitted from the smart glasses. The received data is first pre-processed. Specifically, the video data is divided into frames, the audio data is divided into fixed time units, and the heart rate data is compared with a timestamp. The input is raw data of camera video, audio data, and heart rate data, and the output is a pre-processed dataset.

[1441] Step 3:

[1442] The server applies a face recognition algorithm to pre-processed video data to detect the faces of salespeople. Specifically, it uses OpenCV to extract the position and major feature points of faces for each frame. The input is pre-processed video data, and the output is video data with face detection information added.

[1443] Step 4:

[1444] The server applies a speech recognition algorithm to the audio data, converting it into text data while simultaneously analyzing the characteristics of the speech (tone, pitch, speed, etc.). Specifically, it uses a speech recognition engine with a generative AI model to generate text from the audio data and extract speech features at the same time. The input is pre-processed audio data, and the output consists of text data and speech characteristic data.

[1445] Step 5:

[1446] The server analyzes heart rate data. This includes monitoring heart rate variability in real time and detecting any abnormalities. Specifically, it uses a heart rate analysis algorithm to evaluate heart rate data and detect signs of stress or tension. The input is pre-processed heart rate data, and the output is the analyzed heart rate data.

[1447] Step 6:

[1448] The server integrates the results of video analysis, audio analysis, and heart rate analysis to determine the reliability of the salesperson's words and actions. Specifically, it integrates the analysis results of each data to quantify the probability that the salesperson is lying. The inputs are video data with facial detection information, text data, audio characteristic data, and analyzed heart rate data, and the reliability evaluation results are output.

[1449] Step 7:

[1450] The server sends the evaluation results to the smart glasses and notifies the user. Specifically, it generates a reliability evaluation result as a feedback message and displays it on the smart glasses in real time. The input is the reliability evaluation result, and the feedback message is output. Based on the analysis results displayed on the smart glasses' screen, the user can proceed with the conversation with the salesperson. If a message such as "There is a 70% chance they are lying" is displayed, the user can ask further questions and request a more detailed explanation.

[1451] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1452] This invention is a system that uses a user emotion engine to determine the veracity of a salesperson's words and actions in real time, providing consumers with highly reliable information. This allows consumers to proceed with transactions with confidence.

[1453] Device (smart glasses) side

[1454] The user puts on smart glasses. The smart glasses are equipped with a camera, heart rate sensor, microphone, and emotion engine, and these devices are activated to begin collecting data. The camera captures video of the salesperson in real time, the heart rate sensor monitors the user's heart rate, the microphone records the conversation, and the emotion engine obtains emotional data from the user's voice tone and facial expressions. The collected data is transmitted to a server via Wi-Fi or Bluetooth.

[1455] Server side

[1456] The server receives camera video, audio data, heart rate data, and emotion data sent from the smart glasses. The received data is first pre-processed, which includes the following:

[1457] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the faces of the salespeople.

[1458] The audio data is converted into text data using a speech recognition algorithm.

[1459] Heart rate data is synchronized with video frames over time.

[1460] User sentiment data acquired by the sentiment engine is also preprocessed.

[1461] Next, the server analyzes the pre-processed data. The specific steps are as follows:

[1462] Using a facial expression analysis model, we analyze the subtle facial movements of salespeople and extract indicators of stress and tension.

[1463] We use a speech analysis model to analyze the characteristics of speech data (tone, pitch, speed, etc.).

[1464] It detects abnormal fluctuations (such as sudden increases or irregular heartbeats) from heart rate data.

[1465] We analyze user emotion data and evaluate the user's emotional state.

[1466] These analysis results are integrated to calculate the probability that the salesperson is lying. The analysis results are calculated as a specific percentage, and a message such as "There is a 70% chance that they are lying" is generated. The server sends this feedback message to the smart glasses in real time to notify the user of the result.

[1467] User side

[1468] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[1469] Specific example

[1470] As a concrete example, consider the case of a high-value mortgage contract. The user wears smart glasses and receives an explanation from a salesperson. The smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server, which analyzes the received data in real time. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can request confirmation from the salesperson based on a warning message displayed on the smart glasses. This system allows users to reliably verify important contract details.

[1471] This embodiment allows consumers to quickly and accurately determine the veracity of a salesperson's words and actions, enabling them to proceed with a contract while minimizing risk. Furthermore, by combining it with an emotion engine, a comprehensive analysis that also takes into account the user's emotional state can be performed, enabling even more accurate determination of truthfulness.

[1472] The following describes the processing flow.

[1473] Device (smart glasses) side

[1474] Step 1: Data Acquisition

[1475] The smart glasses activate a camera, heart rate sensor, microphone, and emotion engine. The camera captures video of the salesperson at 30 frames per second, the heart rate sensor monitors the user's heart rate, the microphone records conversations, and the emotion engine extracts emotional data from the user's voice tone and facial expressions.

[1476] Step 2: Send to the server

[1477] The smart glasses transmit collected data to a server in real time. Camera video, heart rate data, audio data, and emotion data are transferred to the server via Wi-Fi or Bluetooth.

[1478] Server side

[1479] Step 3: Data Received

[1480] The server receives data sent from the smart glasses. All data, including camera video, audio, heart rate, and emotion data, is received using a secure communication protocol.

[1481] Step 4: Pretreatment

[1482] The server preprocesses the received data. The specific steps are as follows:

[1483] The camera footage is divided into frames, and a facial recognition algorithm is applied to detect the salesperson's face.

[1484] The audio data is converted to text using a speech recognition algorithm.

[1485] The heart rate data is synchronized with the video frame and timeline.

[1486] The emotional data is formatted appropriately to make it easier to analyze.

[1487] Step 5: Facial Expression Analysis

[1488] The server inputs pre-processed video data into a facial expression analysis model to analyze the subtle movements of the salesperson's face. It then extracts features that serve as indicators of stress and tension.

[1489] Step 6: Voice Analysis

[1490] The server inputs audio data into a speech analysis model, which analyzes speech characteristics such as tone, pitch, and speed. It also detects signs of stress and tension.

[1491] Step 7: Heart Rate Analysis

[1492] The server analyzes heart rate data and detects abnormal fluctuations such as sudden increases or irregularities in the user's heart rate.

[1493] Step 8: Emotion Analysis

[1494] The server analyzes pre-processed emotional data to evaluate the user's emotional state. It identifies emotional responses from the user's tone of voice and facial expressions.

[1495] Step 9: Comprehensive analysis and truth / false determination

[1496] The server integrates the results of facial expression analysis, voice analysis, heart rate analysis, and emotion analysis to make a comprehensive determination of truthfulness. For example, it calculates a percentage, such as "there is a 70% chance that they are lying."

[1497] Step 10: Generate Feedback

[1498] Based on the truth / false determination result, the server generates a feedback message to notify the user. This message, in text format, might state something like, "There is a 70% chance you are lying."

[1499] Step 11: Submitting the analysis results

[1500] The server sends the generated feedback message to the smart glasses. A low-latency communication protocol is used to ensure immediacy.

[1501] Device (smart glasses) side

[1502] Step 12: Display User Interface

[1503] The smart glasses display analysis results received from a server to the user. The analysis results are shown on a display within the user's field of view. Audio feedback is also provided as needed.

[1504] User side

[1505] Step 13: Checking Feedback

[1506] The user reviews the analysis results displayed on the smart glasses. Using the message, "There's a 70% chance they're lying," as a reference, they can ask the salesperson additional questions.

[1507] Step 14: Decision on the contract

[1508] The user will make a final contract decision based on the analysis results. If there are any doubts, they will request further explanations and also review the contract documents.

[1509] Through the above series of steps, a system will be realized that allows consumers to verify the veracity of salespeople's words and actions in real time, enabling them to proceed with contracts with confidence. Furthermore, the introduction of an emotion engine will enable comprehensive analysis that also takes into account the user's emotional state, resulting in even more accurate determination of truthfulness.

[1510] (Example 2)

[1511] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1512] In traditional communication between consumers and salespeople, it was difficult to accurately determine the truthfulness of the salesperson's words and actions, resulting in a high risk of consumers suffering disadvantages. This problem prevented consumers from proceeding with transactions with confidence, especially in financial transactions and high-value contracts. Furthermore, traditional systems struggled to perform comprehensive analysis that took into account the user's emotional state, resulting in low accuracy in determining truthfulness.

[1513] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1514] In this invention, the server includes means for acquiring the user's biometric data, such as heart rate data; means for acquiring the user's emotional state; means for transmitting the data to the server via a communication network; means for pre-processing the received data; and means for integrating the analysis results of the pre-processed data and determining its truthfulness. This makes it possible to quickly and accurately determine the truthfulness of a salesperson's words and actions, allowing consumers to proceed with transactions with peace of mind. Furthermore, by performing a comprehensive analysis that also takes the user's emotional state into consideration, it becomes possible to determine truthfulness with even greater accuracy.

[1515] "Camera footage" refers to video data acquired by a camera installed in a wearable device worn by the user.

[1516] "Voice data" refers to audio information acquired by a microphone installed in a wearable device worn by the user.

[1517] "Heart rate data" refers to data obtained by measuring the user's heart rate, which is a biometric piece of information.

[1518] "Emotional state" refers to the psychological state of a user, obtained by analyzing their voice tone, facial expressions, and other biometric information.

[1519] A "communication network" refers to the infrastructure used to send and receive data, and this includes Wi-Fi, Bluetooth, and other similar technologies.

[1520] "Preprocessing" refers to a series of processes used to convert acquired raw data into an analyzable format, and includes data splitting, synchronization, and normalization.

[1521] "Analysis" is the process of evaluating the characteristics and relationships of pre-processed data using specific algorithms and models.

[1522] "Determining truthfulness" involves calculating the probability of whether a salesperson's statements or actions are true or false based on the analyzed data.

[1523] "Feedback" refers to a means of providing information to notify users of analysis results, and this includes visual displays and audio notifications.

[1524] "Integration" refers to the process of combining different types of pre-processed and analyzed data (such as video, audio, heart rate, and emotions) into a single, comprehensive analysis.

[1525] This invention is a system designed to allow users to proceed with transactions with confidence during interactions with sales representatives. This system uses a wearable device worn by the user (e.g., smart glasses) and a server to determine the veracity of the sales representative's statements in real time. The specific configuration and operation of the entire system are described below.

[1526] Hardware configuration

[1527] Device side (smart glasses)

[1528] Camera: Captures video in real time.

[1529] Mike: Record the conversation.

[1530] Heart rate sensor: Monitors the user's heart rate.

[1531] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[1532] Communication module: Has the function of sending data to a server via Wi-Fi or Bluetooth.

[1533] Server side

[1534] Data receiving module: Receives data transmitted from smart glasses.

[1535] Preprocessing module: Preprocesses the received data.

[1536] Analysis module: Analyzes pre-processed data.

[1537] Feedback generation module: Generates feedback based on the analysis results.

[1538] Communication module: Sends the generated feedback to the smart glasses.

[1539] Data processing and data calculation

[1540] 1. The user puts on the smart glasses and begins the conversation.

[1541] 2. The device (smart glasses) collects camera video, audio data, heart rate data, and emotion data, and transmits them to the server in real time.

[1542] 3. The server divides the received video data into frames and applies a face recognition algorithm to detect the salesperson's face. For example, it uses computer vision technology such as OpenFace.

[1543] 4. The server converts the audio data into text data using a speech recognition algorithm. For example, it uses deep learning-based speech recognition technology such as Kaldi.

[1544] 5. The server uses techniques such as the Short-Term Fourier Transform to synchronize heart rate data and video frames over time.

[1545] 6. The server normalizes the emotional data and evaluates the user's psychological state as numerical data. It uses an emotional analysis model such as EmotionRecognitionNet.

[1546] 7. The server analyzes the pre-processed data and evaluates the characteristics of each data point using facial expression analysis models, voice analysis models, and heart rate analysis models.

[1547] 8. The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[1548] 9. The server generates specific feedback messages based on probability, such as "There is a 70% chance you are lying," and sends these to the smart glasses.

[1549] 10. The user uses the feedback displayed on the smart glasses to guide the conversation. For example, they might ask the salesperson additional questions based on the feedback to verify the accuracy of the information.

[1550] Specific example

[1551] As a concrete example, consider the case of a high-value mortgage contract. When a user wears smart glasses and receives an explanation from a salesperson, the smart glasses' camera captures the salesperson's image, a heart rate sensor monitors the user's heart rate, a microphone records the conversation, and an emotion engine analyzes the user's voice tone and facial expressions. This data is sent to a server in real time and analyzed immediately. If the analysis determines that there is an 80% probability that the salesperson is lying about the details of the benefits, the user can use the warning message displayed on the smart glasses to ask the salesperson for clarification again.

[1552] Example of a prompt

[1553] Please provide a detailed explanation of what a user should do if, during a high-value mortgage contract, smart glasses detect an 80% probability that the salesperson is lying while explaining the benefits.

[1554] The above describes a specific embodiment of the present invention. This system allows users to quickly and accurately determine the truthfulness of a salesperson's words and actions, enabling them to proceed with transactions with confidence. Furthermore, since the analysis also takes into account the user's emotional state, even more accurate determinations are possible.

[1555] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1556] Program processing flow

[1557] Step 1: Device startup and data collection

[1558] input:

[1559] The user puts on smart glasses.

[1560] process:

[1561] The camera on the device (smart glasses) activates and begins recording video in real time.

[1562] The microphone activates and begins recording the conversation.

[1563] The heart rate sensor begins measuring the user's heart rate.

[1564] The emotion engine begins analyzing the user's voice tone and facial expressions in real time.

[1565] output:

[1566] Collected video data, audio data, heart rate data, and emotion data.

[1567] Step 2: Send

[1568] input:

[1569] Collected video data, audio data, heart rate data, and emotion data.

[1570] process:

[1571] The device sends data to the server via Wi-Fi or Bluetooth.

[1572] The communication module compresses the data and sends it to the server via a secure network.

[1573] output:

[1574] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[1575] Step 3: Data Preprocessing

[1576] input:

[1577] Video data, audio data, heart rate data, and emotion data transmitted to the server.

[1578] process:

[1579] The server divides the video data into frames and applies a face recognition algorithm (e.g., OpenFace) to detect the salesperson's face.

[1580] The server converts the audio data into text data using a speech recognition algorithm (e.g., Kaldi).

[1581] The server applies a short-term Fourier transform to synchronize the heart rate data with the video frame.

[1582] Emotional data is normalized, and the user's psychological state is evaluated as numerical data.

[1583] output:

[1584] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[1585] Step 4: Data Analysis

[1586] input:

[1587] Pre-processed video data, text data, synchronized heart rate data, and normalized emotion data.

[1588] process:

[1589] The server uses a facial expression analysis model to analyze the subtle facial movements of the salesperson and extract indicators of stress and tension.

[1590] The server uses a speech analysis model to analyze the characteristics of the speech (tone, pitch, speed, etc.).

[1591] The server detects abnormal fluctuations from the heart rate data.

[1592] The server analyzes emotional data and evaluates the user's emotional state.

[1593] output:

[1594] Salesperson stress and tension indicators, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[1595] Step 5: Truth Determination and Feedback Generation

[1596] input:

[1597] Analyzed stress and tension indices, voice characteristics data, abnormal heart rate variability data, and user emotional state data.

[1598] process:

[1599] The server integrates the analysis results and uses a Naive Bayes classifier to calculate the probability that the salesperson is lying.

[1600] The server generates feedback messages expressing the truthfulness determination result as a specific percentage, such as "There is a 70% chance that they are lying."

[1601] output:

[1602] Feedback message (e.g., "There's a 70% chance you're lying").

[1603] Step 6: Displaying Feedback and User Interaction

[1604] input:

[1605] Feedback message.

[1606] process:

[1607] The device displays a feedback message on the smart glasses' display.

[1608] Voice notifications will also be provided as needed.

[1609] output:

[1610] A feedback message displayed to the user.

[1611] Step 7: User response

[1612] input:

[1613] The displayed feedback message.

[1614] process:

[1615] Users use the feedback messages to ask salespeople additional questions.

[1616] Users will use feedback messages to proceed with the dialogue more carefully.

[1617] output:

[1618] Asking additional questions to the salesperson and facilitating the conversation.

[1619] Through the above processing steps, the system can determine the veracity of the salesperson's statements in real time and provide feedback to the user. This allows the user to proceed with transactions with confidence.

[1620] (Application Example 2)

[1621] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1622] In sales activities, consumers often find it difficult to quickly and accurately determine the truthfulness of a salesperson's words and actions, making it difficult to evaluate the credibility of sales pitches and explanations, which can lead to disadvantages. Furthermore, it is not easy for consumers to understand their own emotional state and make calm judgments. In addition, there is a lack of tools to determine whether what a salesperson is saying is trustworthy, making it difficult to create an environment where consumers can proceed with transactions with peace of mind.

[1623] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for pre-processing, means for applying a face recognition algorithm, means for applying a voice recognition algorithm, means for synchronizing heart rate data over time, and means for performing emotion analysis. This makes it possible to determine the veracity of the salesperson's words and actions in real time and provide consumers with highly reliable information. Furthermore, by analyzing emotion data, it is possible to perform a comprehensive analysis that also takes into account the consumer's emotional state and make even more accurate judgments. This makes it possible to provide consumers with an environment in which they can proceed with transactions with peace of mind.

[1624] "Camera footage" refers to visual information acquired through a camera.

[1625] "Audio data" refers to sound information acquired through a microphone.

[1626] "Heart rate data" refers to numerical information about the user's heart rate measured by a heart rate sensor.

[1627] "Emotional data" refers to data that indicates a user's emotional state, obtained from their tone of voice and facial expressions.

[1628] "Preprocessing" refers to the initial stages of processing that make data easier to analyze.

[1629] A "face recognition algorithm" refers to a computational method for automatically detecting human faces from video data.

[1630] A "speech recognition algorithm" refers to a computational method for converting speech data into text.

[1631] "Synchronizing across timeframes" refers to the process of adjusting different datasets to fit within the same timeframe.

[1632] "Emotional analysis" refers to a technology that analyzes acquired emotional data to evaluate the user's emotional state.

[1633] A "facial expression analysis model" refers to a computational model used to analyze emotions and psychological states from a person's facial expressions.

[1634] A "speech analysis model" refers to a computational model used to analyze the characteristics of speech (such as tone, pitch, and speed).

[1635] "Analysis results" refer to conclusions and evaluations obtained through the analysis of data.

[1636] "Real-time" refers to a state where the time from data acquisition to the display of analysis results is extremely short.

[1637] System Configuration

[1638] This invention provides a system that allows users to wear smart glasses to determine in real time whether a salesperson's words and actions are true or false, thereby providing consumers with reliable information. The system includes smart glasses and a server.

[1639] Device (smart glasses) side

[1640] The user wears smart glasses. The smart glasses have the following hardware and software:

[1641] Camera: Captures real-time footage of the salesperson.

[1642] Heart rate sensor: Monitors the user's heart rate.

[1643] Mike: Record the conversation.

[1644] Emotion Engine: Acquires emotional data from the user's voice tone and facial expressions.

[1645] These devices transmit data to the server via Wi-Fi or Bluetooth.

[1646] Server side

[1647] The server receives data sent from the smart glasses and performs the following processing:

[1648] Preprocessing: Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted to text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames over time, and user emotion data obtained by the emotion engine is also preprocessed.

[1649] Data Analysis: We analyze the subtle facial movements of salespeople using facial expression analysis models (e.g., Dlib, DeepFace), and analyze the characteristics of voice data (tone, pitch, speed, etc.) using voice analysis models. We detect abnormal fluctuations (sudden increases and irregularities in heart rate) from heart rate data and analyze the user's emotional data.

[1650] Results Integration and Feedback: The analysis results are integrated to calculate the probability that the salesperson is lying. The result is calculated as a percentage, and a message such as "There is a 70% chance that they are lying" is generated. This feedback message is sent to the smart glasses in real time to notify the user of the result.

[1651] User side

[1652] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. In this way, users can verify the truthfulness of the salesperson's words and actions and proceed with the contract with confidence.

[1653] Specific example

[1654] As a concrete example, consider the case of purchasing an expensive item (such as jewelry) and determining whether the salesperson's explanation is true. For instance, if a salesperson says, "This diamond is 100% pure and clean-cut," the "Trust Salesperson" application analyzes the information and displays "90% probability of being true" on the smart glasses' display. Based on this information, the customer can proceed with the purchase with confidence.

[1655] Example of a prompt

[1656] A salesperson explains, "This diamond is 100% pure and clean-cut." The customer is wearing smart glasses and using the "Trusted Salesperson" app, which analyzes the conversation, facial expressions, and voice characteristics in real time. The app indicates that there is a 99% probability that the salesperson is telling the truth. What does the customer do next?

[1657] By inputting this prompt into the AI ​​model, it is possible to support the dialogue between sales representatives and customers and facilitate reliable transactions.

[1658] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1659] Step 1:

[1660] The device (smart glasses) acquires camera footage, audio data, heart rate data, and emotion data. The camera captures real-time video of the salesperson, and the microphone records conversations. The heart rate sensor monitors the user's heart rate, and the emotion engine acquires emotion data from the user's voice tone and facial expressions. This data is transmitted to a server via Wi-Fi or Bluetooth.

[1661] Inputs: Camera video, audio data, heart rate data, emotion data

[1662] Output: Data transmission from smart glasses to server

[1663] Step 2:

[1664] The server receives data transmitted from the terminal. The received data is first pre-processed. Camera footage is divided into frames, and a face recognition algorithm (e.g., OpenCV) is applied to detect the faces of salespeople. Audio data is converted into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API). Heart rate data is synchronized with the video frames in time, and emotion data is pre-processed.

[1665] Input: Raw data sent from the terminal

[1666] Output: Preprocessed data (frame-divided video, audio converted to text, synchronized heart rate data, preprocessed emotion data)

[1667] Step 3:

[1668] The server analyzes the pre-processed data. It uses facial expression analysis models (e.g., Dlib, DeepFace) to analyze the subtle facial movements of salespeople, and voice analysis models to analyze the characteristics of voice data (tone, pitch, speed, etc.). It detects abnormal fluctuations (sudden increases or irregularities in heart rate) from heart rate data and analyzes the user's emotional data.

[1669] Input: Preprocessed data

[1670] Output: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[1671] Step 4:

[1672] The server integrates the analysis results and calculates the probability that the salesperson is lying. A statistical model is used to comprehensively evaluate parameters obtained from different analyses in the integration of the analysis results. As a result, messages such as "There is a 70% probability that they are lying" are generated.

[1673] Input: Analysis results (facial expression analysis results, voice analysis results, heart rate analysis results, emotion analysis results)

[1674] Output: Integrated analysis results (probability of lying)

[1675] Step 5:

[1676] The server transmits the integrated analysis results to the smart glasses in real time and notifies the user of the results. The notification is displayed as a message on the smart glasses' display.

[1677] Input: Integrated analysis results

[1678] Output: Feedback message displayed on smart glasses

[1679] Step 6:

[1680] Users interact with salespeople while referring to the analysis results displayed on their smart glasses. For example, if the display shows "There is a 70% chance that they are lying," the user can use that information to ask the salesperson additional questions and request more detailed explanations. This allows users to verify the truthfulness of the salesperson's statements and proceed with the contract with confidence.

[1681] Input: Feedback message displayed on smart glasses

[1682] Output: User actions (additional questions, contract progress)

[1683] Through the steps described above, the present invention provides an environment in which consumers can quickly and accurately determine the truthfulness of a salesperson's words and actions, and proceed with transactions with confidence.

[1684] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1685] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1686] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1687] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1688] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1689] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1690] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1691] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1692] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1693] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1694] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1695] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1696] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1697] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1698] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1699] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1700] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1701] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1702] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1703] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1704] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1705] The following is further disclosed regarding the embodiments described above.

[1706] (Claim 1)

[1707] Means of acquiring camera footage,

[1708] Means for acquiring audio data,

[1709] A means of acquiring heart rate data,

[1710] A means of sending data to a server,

[1711] A means of analyzing the received data,

[1712] A means for displaying the analysis results,

[1713] A system that includes this.

[1714] (Claim 2)

[1715] Means for performing pre-processing,

[1716] A means of applying a facial recognition algorithm,

[1717] Means for applying speech recognition algorithms,

[1718] A method for synchronizing heart rate data over time,

[1719] The system according to claim 1, including the following:

[1720] (Claim 3)

[1721] Methods for analysis using facial expression analysis models,

[1722] Methods for analysis using speech analysis models,

[1723] A means of analyzing heart rate data,

[1724] The system according to claim 1, including the following:

[1725] "Example 1"

[1726] (Claim 1)

[1727] Means of acquiring camera footage,

[1728] Means for acquiring audio data,

[1729] A means of acquiring heart rate data,

[1730] A means of sending data to a server,

[1731] A means for preprocessing the received data,

[1732] A means for analyzing preprocessed data,

[1733] A means for displaying the analysis results,

[1734] A system that includes this.

[1735] (Claim 2)

[1736] A means of dividing camera footage into frames,

[1737] A means of applying a facial recognition algorithm,

[1738] A means of applying a speech recognition algorithm that converts audio data into text data,

[1739] A method for synchronizing heart rate data over time,

[1740] The system according to claim 1, including the following:

[1741] (Claim 3)

[1742] Methods for analysis using facial expression analysis models,

[1743] A method for analyzing tone, pitch, and speed using a speech analysis model,

[1744] A means of analyzing heart rate data,

[1745] A method for integrating multiple analysis results to calculate the probability that a salesperson is lying,

[1746] A means of notifying the user of the analysis results in real time,

[1747] The system according to claim 1, including the following:

[1748] "Application Example 1"

[1749] (Claim 1)

[1750] Means of acquiring camera footage,

[1751] Means for acquiring audio data,

[1752] A means of acquiring heart rate data,

[1753] A means of sending data to a server,

[1754] A means of analyzing the received data,

[1755] A means for displaying the analysis results,

[1756] A means of notifying users of analysis results in real time,

[1757] A system that includes this.

[1758] (Claim 2)

[1759] Means for performing pre-processing,

[1760] A means of applying a facial recognition algorithm,

[1761] Means for applying speech recognition algorithms,

[1762] A method for synchronizing heart rate data over time,

[1763] A means for determining reliability by integrating video analysis, audio analysis, and heart rate analysis,

[1764] The system according to claim 1, including the following:

[1765] (Claim 3)

[1766] Methods for analysis using facial expression analysis models,

[1767] Methods for analysis using speech analysis models,

[1768] A means of analyzing heart rate data,

[1769] A method for calculating the reliability evaluation result as a probability,

[1770] A means of displaying feedback messages on the smart glasses' display,

[1771] The system according to claim 1, including the following:

[1772] "Example 2 of combining an emotion engine"

[1773] (Claim 1)

[1774] Using a wearable device worn by the user,

[1775] Means of acquiring camera footage,

[1776] Means for acquiring audio data,

[1777] A means of acquiring the user's biometric data, specifically heart rate data,

[1778] A means of obtaining the user's emotional state,

[1779] A means of sending data to a server via a communication network,

[1780] A means of pre-processing the data received on the server side,

[1781] A means for analyzing preprocessed data,

[1782] A means of providing feedback on the analysis results to the user,

[1783] A system that includes this.

[1784] (Claim 2)

[1785] A means of dividing the received video data into frames and applying a face recognition algorithm,

[1786] A means of ...

Claims

1. A means for acquiring camera images, which are image data obtained from a device that converts real-world visual information into digital data and records it, A means for acquiring audio data, which is data recorded by converting the waveform of sound into a digital signal, A means for measuring the user's heartbeat and acquiring heart rate data, which is data showing the heart rate at regular intervals, A means of sending data to a server, A means of analyzing the received data, A means for displaying the analysis results, A system that includes this.

2. Means for performing pre-processing, A means of applying a facial recognition algorithm, Means for applying speech recognition algorithms, A method for synchronizing heart rate data over time, The system according to claim 1, including the following:

3. Methods for analysis using facial expression analysis models, Methods for analysis using speech analysis models, A means of analyzing heart rate data, The system according to claim 1, including the following:

Images