system

Abstract

We provide the system. [Solution] Means for receiving user audio and video data, A means for analyzing received audio and video data to detect features related to user manners and etiquette, A means for generating user-selectable character feedback based on analysis results, A means of notifying the user's device of the generated feedback, Means to anonymize or delete data in order to protect users' personal information, A system that includes this.

Description

Technical Field

[0004] , , , ,

[0005] , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0006] "Audio data" refers to data obtained from users in the form of digital audio information.

[0007] "Video data" refers to data obtained by converting a user's visual information into a digital format.

[0008] "Analysis" is the process of processing acquired audio and video data to recognize specific features and patterns.

[0009] "Features" are pieces of information derived from analyzed data that indicate unique properties or tendencies related to user behavior and state.

[0010] "Feedback" refers to the information, such as suggestions and evaluations, provided to users based on the analysis results.

[0011] A "character" is a virtual personality or avatar used to convey feedback through a user interface.

[0012] A "terminal" is a device used by a user to perform operations or receive information.

[0013] "Protection of personal information" refers to the process of properly managing data and preventing the disclosure or access of unnecessary information in order to protect user privacy.

[0014] "Anonymization" is a method of deleting or converting personally identifiable information so that the data cannot be associated with an individual.

[0015] "Data" is a general term for information including voice and video that is required to generate feedback.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a system for facilitating feedback on user manners and etiquette, providing feedback to the user through selectable characters via the analysis of audio and video data. The system is primarily configured and operates as follows:

[0038] First, the user launches the application on their device. This application has the ability to capture the user's voice and video in real time using the camera and microphone. The user starts capturing data when they need specific feedback.

[0039] The device transmits captured audio and video data to a server over the network. The server analyzes this data using advanced image and speech recognition algorithms. The analysis identifies features related to user etiquette, such as the detection of nasal hairs or signs of bad breath.

[0040] Based on the results of this analysis, the server generates appropriate feedback through the character selected by the user. This feedback consists of a friendly character's way of speaking, such as, "You have nose hairs sticking out, fix it."

[0041] Next, the generated feedback is sent back to the device and notified to the user visually or audibly. The device uses audio playback and character animation to naturally convey the feedback to the user. This notification is seamless and interactive so as not to interfere with the user's feedback reception.

[0042] Finally, the server strives to protect users' personal information by anonymizing or deleting data. This allows users to continue using the feedback service without worrying about their privacy.

[0043] As a concrete example, imagine a scenario where a young business professional is preparing a presentation. The user launches the app on their device, turns on the camera and microphone, and captures themselves. While the user practices the presentation, the server analyzes feedback on the user's voice clarity and gestures, and points out areas for improvement through an appropriate character. The user receives this feedback immediately and can improve their performance.

[0044] In this way, the system of this invention provides enjoyable and effective support for self-improvement in the user's daily life and business situations.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The user launches a dedicated application on their device. The application, with the user's consent, enables the camera and microphone. The user selects the option to start data capture when they want to receive feedback.

[0048] Step 2:

[0049] The device captures the user's audio and video data. This process is performed in real time, and the captured data is temporarily stored on the device. Here, basic processing such as noise reduction is performed to optimize the data quality.

[0050] Step 3:

[0051] The terminal processes the audio and video data and transmits it to the server via the network. The data is encrypted using secure communication technology to protect privacy.

[0052] Step 4:

[0053] The server analyzes the received audio and video data. The machine learning algorithms used here are optimized to detect features related to predefined manners and etiquette. For example, the server analyzes facial images for the presence or absence of nose hairs, and audio for speech clarity.

[0054] Step 5:

[0055] The server generates user-appropriate feedback based on the analysis results. This feedback is tailored to the personality of the character selected by the user. For example, it may generate gentle criticism or humorous comments.

[0056] Step 6:

[0057] The server sends generated feedback to the device. This feedback can be in the form of voice messages, text messages, or animations in which the avatar moves.

[0058] Step 7:

[0059] The device presents feedback from the server to the user. Notifications are provided in the user's preferred visual or auditory form. The device provides feedback to the user in a more understandable way through an interactive user interface.

[0060] Step 8:

[0061] The server protects the user's personal information. After all processing is complete, the server anonymizes the data or, if necessary, completely deletes it. This ensures that users can continue to use the service while maintaining their privacy.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] Obtaining immediate and effective feedback on etiquette and manners in users' daily lives and work is difficult. Traditional methods require third-party criticism or individual training, which are time-consuming and costly. Furthermore, the feedback is often general and lacks specificity tailored to individual situations. As a result, users miss opportunities to immediately improve their own behavior.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for receiving user voice and video data, means for analyzing the received data to detect features related to the user's behavior, and means for generating feedback to the user through a selected character based on the analysis results. This allows the user to receive specific and personalized feedback on their behavior in real time and take immediate corrective action.

[0067] A "user" refers to an individual who uses the system to receive feedback.

[0068] "Voice data" refers to digital information collected and recorded by a device such as a microphone when a user speaks.

[0069] "Video data" refers to a digital representation of a user's visual information captured by a camera or other device.

[0070] "Analysis" refers to the process of using specific algorithms to identify features related to etiquette and manners in received audio and video data.

[0071] "Features" refer to characteristics related to etiquette and manners extracted from audio and video data, and are indicators used for generating feedback.

[0072] "Feedback" refers to responses or suggestions that provide users with information to improve their behavior, which can help users improve their etiquette and manners.

[0073] A "character" refers to a visual or auditory agent used to convey feedback to the user, providing a sense of familiarity.

[0074] "Anonymization" refers to the process of removing personally identifiable information from data so that users' personal information cannot be identified.

[0075] "Deletion" refers to the process of completely removing data, including a user's personal information, and is one way of protecting privacy.

[0076] "Real-time" refers to a processing method where data acquisition, analysis, and feedback provision are all performed instantly.

[0077] "Interactive" refers to a process in which information and feedback are provided based on interaction with the user, indicating the ability to respond to user reactions.

[0078] This invention describes a specific implementation of a system for providing feedback on user behavior. This system aims to provide feedback tailored to the individual circumstances of the user.

[0079] First, the user launches a dedicated application on their device. The device uses its camera and microphone to capture the user's video and audio data in real time. This data is then transmitted to a server via the network.

[0080] Next, the server analyzes the received data using advanced data analysis algorithms. Here, it uses image recognition and speech recognition technologies powered by machine learning models to identify features related to user behavior. Specifically, this includes detecting nasal hairs and analyzing speech clarity. This process also includes models for facial recognition and speech tone analysis.

[0081] Based on the analysis results, the server generates feedback through the character selected by the user. This feedback uses a friendly tone and animations to provide the user with specific suggestions for improving their behavior.

[0082] The generated feedback is notified to the user by the device. The notification is provided visually through character animations and audibly through audio feedback. This allows the user to receive feedback smoothly and immediately have the opportunity to improve their actions.

[0083] As a concrete example, consider a young business professional preparing a presentation. The user launches the app on their device and turns on the camera and microphone. The server analyzes the user's voice and gestures and provides suggestions for improvement through a friendly character. This feedback allows the user to instantly improve their presentation skills.

[0084] An example of a prompt message might be, "Select a character to point out areas for improvement in your gestures during your business presentation, and generate appropriate feedback." By utilizing such prompt messages, the system can generate feedback tailored to the user's needs.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The user launches a dedicated application on their device. The input is the user's command to launch the application. The output is the application activating the camera and microphone and entering capture mode. The camera begins collecting video data, and the microphone begins collecting audio data.

[0088] Step 2:

[0089] The terminal transmits captured audio and video data to the server in real time. The input is raw data acquired from the terminal's camera and microphone. The specific operation of this step is to compress the data and transmit it over the internet. The output is the server, which receives the encoded data over the network.

[0090] Step 3:

[0091] The server applies an AI model to analyze the received audio and video data. It receives encoded data from the terminal as input. By applying algorithms such as facial recognition and voice tone analysis to this data, it extracts features related to etiquette. The output includes information such as nasal hair detection and pronunciation clarity.

[0092] Step 4:

[0093] The server generates feedback for the user based on the analysis results. It uses the feature information obtained from the analysis as input. It customizes the voice and actions of the selected character to create a friendly feedback message. The output includes the generated feedback message and character animation data.

[0094] Step 5:

[0095] The server sends the generated feedback back to the terminal. It uses the generated feedback data as input. The specific operation involves encoding this data in a format suitable for the terminal and then sending it. The output is feedback data that can be displayed on the terminal.

[0096] Step 6:

[0097] The device notifies the user of the feedback data it has received. Its input is feedback sent from the server. Its specific actions include playing the feedback message aloud and displaying character animations. As output, the user is provided with visual and auditory feedback.

[0098] Step 7:

[0099] The server anonymizes or deletes processed data to protect users' personal information. The inputs used are the raw data to be analyzed and the intermediate data used to generate feedback. By anonymizing or completely deleting these, the output remains privacy-protected.

[0100] (Application Example 1)

[0101] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0102] In today's commercial environment, improving employee customer service attitudes and behavior is crucial for increasing customer satisfaction and enhancing a store's reputation. However, providing real-time feedback on individual employee etiquette and manners is difficult. To address this challenge, an effective system is needed that allows employees to immediately improve their own customer service attitudes.

[0103] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0104] In this invention, the server includes means for acquiring user voice and video information, means for analyzing the acquired voice and video information and evaluating features related to the user's manners and etiquette, and means for generating a response from a character with selectable displays for the user based on the analysis results. This enables employees to receive real-time feedback on their attitude and behavior during customer service, make immediate improvements, and enhance customer satisfaction.

[0105] A "user" refers to a person who uses the system and receives feedback from it.

[0106] "Voice information" refers to data related to the voice and sounds obtained from the user.

[0107] "Visual information" refers to visual data that captures the user's appearance and actions.

[0108] "Means of acquisition" refers to devices and methods for electronically capturing audio and video information.

[0109] "Means of analysis" refers to technologies for processing acquired audio and video information and extracting characteristics related to the user's attitude and behavior.

[0110] "Etiquette and manners" refers to appropriate behavior and communication styles that conform to social norms.

[0111] "Means of evaluation" refers to a mechanism that makes judgments based on criteria derived from the characteristics obtained from the analyzed data.

[0112] A "character with a display" refers to a character that is visually or audibly represented to convey feedback to the user.

[0113] "Means of generating responses" refers to technologies for constructing feedback to be given to users based on evaluation results.

[0114] The system of this invention acquires audio and video information, analyzes it, and provides feedback to the user. Specifically, when an employee wears a device such as smart glasses in a store and interacts with customers, their audio and video are captured in real time. The acquired data is transmitted to a server via a network and analyzed using advanced image recognition and speech recognition algorithms. The server evaluates features related to the user's manners and etiquette and generates feedback based on the results.

[0115] This system processes and analyzes data using software such as OpenCV and Google® Speech Recognition. The analysis results are communicated to the user through a character with selectable displays on the user's device. Depending on the user's choice, this character provides feedback in a user-friendly format through either sight or sound.

[0116] For example, when a new employee at a cafe serves customers for the first time, this system can be used to quickly identify areas for improvement, such as not smiling enough, allowing them to immediately improve their attitude and ultimately increase customer satisfaction.

[0117] An example of a prompt message might be, "Capture the current customer service interaction through your smart glasses and evaluate the frequency of smiles during the interaction. Select a character and provide feedback such as 'Smile more.'" This allows users to understand and implement specific ways to improve their own behavior.

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The device uses the smart glasses' camera and microphone to acquire the user's voice and video information in real time. The input to this process is the user's raw voice and video, and the output is digitized voice and video data. This digital data is then transmitted to the next step.

[0121] Step 2:

[0122] The terminal transmits the acquired audio and video information to the server via the network. Data processing during this transmission process includes data compression and format conversion. The output of this step is audio and video data converted into an analyzable format.

[0123] Step 3:

[0124] The server analyzes the received audio and video data using image recognition and speech recognition algorithms. It uses OpenCV to detect facial expressions and gestures from the video data, and Google Speech Recognition to recognize the audio content as text data. The output of this step is data that quantifies or categorizes features related to the user's manners and etiquette.

[0125] Step 4:

[0126] The server generates feedback for the user based on the analysis results. This feedback involves using a generative AI model to synthesize appropriate responses from prompts using a character. The input to this procedure is the analyzed feature data, and the output is text or audio representation as feedback.

[0127] Step 5:

[0128] The server sends the generated feedback to the terminal. The terminal then plays the feedback visually or audibly through a selected character to present it to the user. The output of this process is the feedback information that the user receives directly. This points out attitudes and behaviors that need improvement and includes specific suggestions for improvement.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention combines an emotion engine with a system that analyzes user voice and video data and provides feedback from selectable characters to improve manners and etiquette. This makes it possible to adjust the feedback according to the user's emotional state.

[0131] First, the user initiates the capture of audio and video data via a dedicated application on their device. The data is sent from the device to a server, which extracts features related to the user's manners and etiquette through image recognition and speech recognition.

[0132] Furthermore, in this invention, an emotion engine embedded in the server analyzes the user's voice tone and facial expressions to recognize emotions in real time. The emotion engine extracts emotional states such as joy, anger, sadness, and surprise, and incorporates this data into the feedback generation process.

[0133] Feedback is delivered in different tones and expressions depending on the character selected by the user. The server adjusts the content and tone of the feedback based on emotional data obtained from the emotion engine. For example, if the server determines that the user is stressed, the feedback will be adjusted to be delivered in gentler language.

[0134] The generated feedback is notified to the user through the device. The notification is presented as a text message, voice message, or as the movement and voice of an animated character. Users can also review the feedback on their device and customize the type and format of the feedback to their preference.

[0135] From the perspective of protecting personal information, the server has the capability to ultimately anonymize or delete data. This allows users to continue receiving feedback in a privacy-protected environment.

[0136] As a concrete example, consider a scenario where a user is practicing conversations at work. The terminal records the user's speech, and the server analyzes the content, tone, and facial expressions of the conversation. The emotion engine detects when the user is feeling nervous and provides relaxing feedback tailored to that emotion. The user receives this feedback and can effectively improve their communication skills. In this way, the system of the present invention enables more advanced self-improvement support than conventional methods.

[0137] The following describes the processing flow.

[0138] Step 1:

[0139] The user launches a dedicated application on their device. Within the application, the user selects options corresponding to the situations in which they want feedback, and, if necessary, sets permissions for camera and microphone use.

[0140] Step 2:

[0141] The device captures the user's audio and video data in real time. The captured data is temporarily stored on the device, and initial processing such as noise reduction and image quality improvement is performed on the data.

[0142] Step 3:

[0143] The terminal transmits processed audio and video data to the server using a secure communication method. During this process, the data is encrypted to prevent unauthorized access by third parties.

[0144] Step 4:

[0145] A machine learning model is applied to analyze the audio and video data received by the server. The server detects characteristics related to the user's manners and etiquette and extracts specific points of criticism.

[0146] Step 5:

[0147] The server uses an emotion engine to analyze the user's emotional state from their voice tone and video. For example, it identifies emotions such as joy, tension, and anger from the user's facial expressions and voice tone.

[0148] Step 6:

[0149] The server uses analysis results and emotional data to generate personalized feedback for the user. The feedback is structured to match the personality of the character selected by the user, with appropriate tone and content. The emphasis and expression of the feedback are adjusted according to the emotional state.

[0150] Step 7:

[0151] The server generates feedback and sends it to the device. The feedback is provided in a format specified by the user, such as synthesized speech, text messages, or animations.

[0152] Step 8:

[0153] The device provides feedback to the user. The user can review the feedback and pay attention to the etiquette improvements recommended by the system. Users can also customize the content and format of the feedback according to their preferences.

[0154] Step 9:

[0155] After the server finishes processing, it anonymizes or deletes user data. Data containing personal information, especially sentiment data, is managed appropriately to ensure privacy. This allows for the provision of continuous feedback while protecting user privacy.

[0156] (Example 2)

[0157] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0158] In modern society, where communication and manners are increasingly important, there is a need for support in helping individuals effectively improve their manners and etiquette. However, traditional methods have struggled to provide appropriate feedback in real time, taking into account individual emotional states. Furthermore, there are insufficient methods to provide personalized feedback while protecting user privacy. To address these challenges, there is a need to develop a system that efficiently and safely improves individuals' communication skills.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0160] In this invention, the server includes means for analyzing user information data and detecting features, means for utilizing emotion recognition data for feedback generation, and means for anonymizing or deleting data. This enables advanced self-improvement support tailored to the user's emotional state.

[0161] "Information data" refers to input data, including user voice and video, which the system analyzes to identify features and utilize for feedback generation.

[0162] "Analysis" refers to the process of extracting and identifying user-related characteristics from information data, thereby obtaining data that serves as the basis for feedback.

[0163] "Characteristics" refer to attributes and patterns identified from user information data, representing behaviors and states related to manners and etiquette.

[0164] A "representation model" refers to a selectable character or method of expression used to set the style and tone of the feedback and responses provided to the user.

[0165] A "communication terminal" refers to a device used by a user to receive feedback and notifications generated by the system.

[0166] "Machine learning technology" refers to algorithms and models used in data analysis, which automatically extract features from informational data and contribute to feedback generation.

[0167] "Anonymization" is the process of processing data in a way that makes it impossible to identify personal information, with the aim of protecting privacy.

[0168] This invention is a system that analyzes voice and video data and provides feedback related to manners and etiquette in order to improve the individual communication skills of users. The specific forms in which this system is implemented are described below.

[0169] First, users use a dedicated application on their device to record audio and video data of everyday conversations and specific situations. This data is temporarily stored in random-access memory or storage devices before being sent to a server via a secure communication protocol. The application installed on the device has recording and shooting functions, efficiently collecting user data.

[0170] The server runs image recognition and speech recognition programs to analyze the received data. For example, machine learning libraries such as TENSORFLOW® and PyTorch are used, allowing the server to recognize the user's voice tone and facial expressions. Based on this, an emotion engine is activated to extract the user's emotional state in real time. Emotional states are classified into categories such as joy, anger, sadness, and surprise.

[0171] Furthermore, the server uses this data to generate different feedback depending on the user's chosen representation model (character). The feedback is adjusted according to the user's emotional state; for example, if the server determines that the user is stressed, it will respond using calmer language.

[0172] Users are notified of this generated feedback on their device in visual or auditory form. Notification formats include text messages, voice messages, and the voice of an animated character, and users can choose from these options. Users can also review the feedback and change settings via the application if necessary.

[0173] As an example of how this system works, imagine a scenario where a user is practicing a presentation at work. The user records their speech on their device, and the data is sent to the server. The server analyzes the tone of the conversation and facial expressions to identify when the user is feeling nervous. Then, it generates relaxing feedback and notifies the user on their device.

[0174] A concrete example of a prompt would be, "I would like feedback on how my speaking style and facial expressions are being perceived during presentation practice at work." In this way, users can effectively improve their skills through the system.

[0175] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0176] Step 1:

[0177] The user initiates the capture of audio and video data using a dedicated application. During this process, the user records their speech and facial expressions using the device's camera and microphone. The input is the user's real-time audio and video data, which is captured directly. The output is the captured audio and video data.

[0178] Step 2:

[0179] The terminal sends the captured audio and video data to the server using a secure protocol. SSL / TLS encrypted communication is used for this transmission. The input is the data captured in step 1, which is converted into a format that the server can process and then sent. The output is the completion of data transmission to the server.

[0180] Step 3:

[0181] The server analyzes the received data and uses an image recognition algorithm to analyze the user's facial expressions. In addition, it uses speech recognition software to analyze the tone and intonation of the voice. The input is the audio and video data sent to the server, and the output is the analyzed user characteristic data.

[0182] Step 4:

[0183] The emotion engine embedded in the server identifies the user's emotional state based on the analyzed data. This identification process uses a machine learning model to extract certain emotional categories. The input is the feature data obtained in step 3, and the output is the identified emotional state data.

[0184] Step 5:

[0185] The server uses collected emotional state data to generate personalized feedback based on the user's selected expression model. This feedback includes adjustments to content and tone according to the emotion. The input is emotional state data and the user's selected expression model, and the output is the generated feedback message.

[0186] Step 6:

[0187] The terminal receives feedback sent from the server and notifies the user. Notifications can take the form of text messages, voice messages, or animated visuals. The input is feedback data from the server, and the output is various forms of feedback presented to the user.

[0188] Step 7:

[0189] Users can review the feedback and, if necessary, modify the format and representation model of the feedback through the application. This allows users to explore more effective feedback patterns. The input is the presented feedback and the user's selection, and the output is the new feedback setting adjusted by the user.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] In traditional customer service, improving employee manners and service skills is crucial, but there have been limited means of providing objective, real-time feedback. Furthermore, because feedback is often fixed, it's difficult to provide appropriate advice tailored to the individual employee's emotional state. This invention aims to solve these problems and effectively improve employee customer service skills and customer satisfaction.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for receiving user voice and video data, means for an emotion engine that analyzes the user's emotional state and adjusts feedback based on the analysis results, and means for anonymizing or deleting data to protect the user's personal information. This enables the provision of real-time feedback that takes the user's emotions into consideration.

[0195] "User voice data" refers to digital signals containing information spoken by the user, and is the acoustic data that will be analyzed.

[0196] "Video data" refers to digital signals containing information that visually captures the user's movements and facial expressions, and is the visual data that is analyzed.

[0197] "Analysis means" refers to a device or system that processes received audio and video data and executes computational techniques or algorithms to detect features and patterns.

[0198] "Characteristics of manners and etiquette" refer to traits that represent the socially expected norms and politeness regarding a user's behavior and speech.

[0199] A "feedback generation method" is a system that creates information that provides appropriate advice and guidance to the user based on the analyzed data.

[0200] "Emotional state" refers to elements that indicate the user's mental or psychological state, and is inferred from factors such as voice tone and facial expressions.

[0201] An "emotion engine" is a hardware or software module that recognizes emotional states from audio and video data and adjusts appropriate feedback accordingly.

[0202] "Anonymization" is a method of processing data in a way that makes it impossible to identify individuals, and it is a technology for protecting privacy.

[0203] "Means of deletion" refer to the technologies and processes used to erase and make inaccessible stored data.

[0204] This invention is a system primarily aimed at improving manners in customer service at physical stores. The system utilizes smart glasses worn by the user to transmit audio and video data to a server in real time. The smart glasses are equipped with a camera and microphone to capture the user's voice and facial expressions.

[0205] The server uses the Google Cloud Speech-to-Text API to convert audio data into text format and the OpenCV library to analyze facial expressions from video data. This data is then analyzed by an emotion engine using IBM Watson® Tone Analyzer to identify emotional states and extract features related to the user's manners and etiquette.

[0206] Based on this, the server uses a Python program to generate feedback appropriate to the selected character. The feedback is displayed in real time on the smart glasses or notified to the user via voice. The tone and content of this feedback are adjusted according to the user's emotional state.

[0207] For example, if it is determined that a store employee is feeling anxious while serving a customer, a message such as "Please relax a little more and smile" will be provided as feedback. Furthermore, from a personal information protection standpoint, the data is anonymized on the server and managed appropriately.

[0208] An example of inputting prompts into a generative AI model is, "Generate feedback to help a store employee relax if they are feeling nervous while serving a customer." This allows the system to effectively improve employees' customer service skills through accurate feedback.

[0209] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0210] Step 1:

[0211] The terminal (smart glasses) captures the user's audio and video data in real time during customer service. The captured data is transmitted to the server via wireless communication. The input for this step is the user's real-time audio and video, and the output is audio and video data in digital format sent to the server.

[0212] Step 2:

[0213] The server analyzes the received audio data using the Google Cloud Speech-to-Text API and converts the audio signal into text data. The input is audio data, and the converted output is text data. The server uses this data to understand what the user is saying.

[0214] Step 3:

[0215] The server analyzes video data using the OpenCV library to analyze the user's facial expressions. The input is video data, and the output is feature data related to the user's facial expressions. This process allows for obtaining real-time information about the user's facial expressions.

[0216] Step 4:

[0217] The server uses IBM Watson Tone Analyzer to identify the emotional state based on the analysis of voice and facial expressions. The input for this step is voice-text data and facial expression feature data, and the output is data indicating the user's emotional state. The server uses the obtained emotional state to adjust the feedback.

[0218] Step 5:

[0219] The server generates feedback using a Python program based on the analyzed emotional state and manners characteristics. The input for this step is emotional state data and manners characteristic data, and the output is the generated feedback message. The feedback is expressed differently depending on the character selected by the user.

[0220] Step 6:

[0221] The device notifies the user of the generated feedback. The notification is displayed on the smart glasses' screen or played as audio. The input for this step is the generated feedback message, and the output is the feedback notification the user receives. This helps the user improve their customer service skills in real time.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention is a system for facilitating feedback on user manners and etiquette, providing feedback to the user through selectable characters via the analysis of audio and video data. The system is primarily configured and operates as follows:

[0239] First, the user launches the application on their device. This application has the ability to capture the user's voice and video in real time using the camera and microphone. The user starts capturing data when they need specific feedback.

[0240] The device transmits captured audio and video data to a server over the network. The server analyzes this data using advanced image and speech recognition algorithms. The analysis identifies features related to user etiquette, such as the detection of nasal hairs or signs of bad breath.

[0241] Based on the results of this analysis, the server generates appropriate feedback through the character selected by the user. This feedback consists of a friendly character's way of speaking, such as, "You have nose hairs sticking out, fix it."

[0242] Next, the generated feedback is sent back to the device and notified to the user visually or audibly. The device uses audio playback and character animation to naturally convey the feedback to the user. This notification is seamless and interactive so as not to interfere with the user's feedback reception.

[0243] Finally, the server strives to protect users' personal information by anonymizing or deleting data. This allows users to continue using the feedback service without worrying about their privacy.

[0244] As a concrete example, imagine a scenario where a young business professional is preparing a presentation. The user launches the app on their device, turns on the camera and microphone, and captures themselves. While the user practices the presentation, the server analyzes feedback on the user's voice clarity and gestures, and points out areas for improvement through an appropriate character. The user receives this feedback immediately and can improve their performance.

[0245] In this way, the system of this invention provides enjoyable and effective support for self-improvement in the user's daily life and business situations.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The user launches a dedicated application on their device. The application, with the user's consent, enables the camera and microphone. The user selects the option to start data capture when they want to receive feedback.

[0249] Step 2:

[0250] The device captures the user's audio and video data. This process is performed in real time, and the captured data is temporarily stored on the device. Here, basic processing such as noise reduction is performed to optimize the data quality.

[0251] Step 3:

[0252] The terminal processes the audio and video data and transmits it to the server via the network. The data is encrypted using secure communication technology to protect privacy.

[0253] Step 4:

[0254] The server analyzes the received audio and video data. The machine learning algorithms used here are optimized to detect features related to predefined manners and etiquette. For example, the server analyzes facial images for the presence or absence of nose hairs, and audio for speech clarity.

[0255] Step 5:

[0256] The server generates user-appropriate feedback based on the analysis results. This feedback is tailored to the personality of the character selected by the user. For example, it may generate gentle criticism or humorous comments.

[0257] Step 6:

[0258] The server sends generated feedback to the device. This feedback can be in the form of voice messages, text messages, or animations in which the avatar moves.

[0259] Step 7:

[0260] The device presents feedback from the server to the user. Notifications are provided in the user's preferred visual or auditory form. The device provides feedback to the user in a more understandable way through an interactive user interface.

[0261] Step 8:

[0262] The server protects the user's personal information. After all processing is complete, the server anonymizes the data or, if necessary, completely deletes it. This ensures that users can continue to use the service while maintaining their privacy.

[0263] (Example 1)

[0264] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] Obtaining immediate and effective feedback on etiquette and manners in users' daily lives and work is difficult. Traditional methods require third-party criticism or individual training, which are time-consuming and costly. Furthermore, the feedback is often general and lacks specificity tailored to individual situations. As a result, users miss opportunities to immediately improve their own behavior.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0267] In this invention, the server includes means for receiving user voice and video data, means for analyzing the received data to detect features related to the user's behavior, and means for generating feedback to the user through a selected character based on the analysis results. This allows the user to receive specific and personalized feedback on their behavior in real time and take immediate corrective action.

[0268] A "user" refers to an individual who uses the system to receive feedback.

[0269] "Voice data" refers to digital information collected and recorded by a device such as a microphone when a user speaks.

[0270] "Video data" refers to a digital representation of a user's visual information captured by a camera or other device.

[0271] "Analysis" refers to the process of using specific algorithms to identify features related to etiquette and manners in received audio and video data.

[0272] "Features" refer to characteristics related to etiquette and manners extracted from audio and video data, and are indicators used for generating feedback.

[0273] "Feedback" refers to responses or suggestions that provide users with information to improve their behavior, which can help users improve their etiquette and manners.

[0274] A "character" refers to a visual or auditory agent used to convey feedback to the user, providing a sense of familiarity.

[0275] "Anonymization" refers to the process of removing personally identifiable information from data so that users' personal information cannot be identified.

[0276] "Deletion" refers to the process of completely removing data, including a user's personal information, and is one way of protecting privacy.

[0277] "Real-time" refers to a processing method where data acquisition, analysis, and feedback provision are all performed instantly.

[0278] "Interactive" refers to a process in which information and feedback are provided based on interaction with the user, indicating the ability to respond to user reactions.

[0279] This invention describes a specific implementation of a system for providing feedback on user behavior. This system aims to provide feedback tailored to the individual circumstances of the user.

[0280] First, the user launches a dedicated application on their device. The device uses its camera and microphone to capture the user's video and audio data in real time. This data is then transmitted to a server via the network.

[0281] Next, the server analyzes the received data using advanced data analysis algorithms. Here, it uses image recognition and speech recognition technologies powered by machine learning models to identify features related to user behavior. Specifically, this includes detecting nasal hairs and analyzing speech clarity. This process also includes models for facial recognition and speech tone analysis.

[0282] Based on the analysis results, the server generates feedback through the character selected by the user. This feedback uses a friendly tone and animations to provide the user with specific suggestions for improving their behavior.

[0283] The generated feedback is notified to the user by the terminal. The notification is provided in the form of visually animating characters and aurally playing voice feedback. This enables the user to smoothly receive feedback and obtain an opportunity to immediately improve their actions.

[0284] As a specific example, consider a young business professional preparing for a presentation. The user launches an app on the terminal and turns on the camera and microphone. The server analyzes the user's voice and gestures and indicates improvement points through friendly characters. With this feedback, the user can immediately improve their presentation skills.

[0285] Examples of prompt sentences include "Please select a character that points out improvement points in gestures during a business presentation and generate appropriate feedback." By utilizing such prompt sentences, the system can generate feedback according to the user's needs.

[0286] The flow of the specific process in Example 1 will be described using FIG. 11.

[0287] Step 1:

[0288] The user launches a dedicated application on the terminal. As input, it receives an app launch instruction by the user's operation. As output, the app activates the camera and microphone and enters the capture mode. The camera starts collecting video data, and the microphone starts collecting audio data.

[0289] Step 2:

[0290] The terminal transmits captured audio and video data to the server in real time. The input is raw data acquired from the terminal's camera and microphone. The specific operation of this step is to compress the data and transmit it over the internet. The output is the server, which receives the encoded data over the network.

[0291] Step 3:

[0292] The server applies an AI model to analyze the received audio and video data. It receives encoded data from the terminal as input. By applying algorithms such as facial recognition and voice tone analysis to this data, it extracts features related to etiquette. The output includes information such as nasal hair detection and pronunciation clarity.

[0293] Step 4:

[0294] The server generates feedback for the user based on the analysis results. It uses the feature information obtained from the analysis as input. It customizes the voice and actions of the selected character to create a friendly feedback message. The output includes the generated feedback message and character animation data.

[0295] Step 5:

[0296] The server sends the generated feedback back to the terminal. It uses the generated feedback data as input. The specific operation involves encoding this data in a format suitable for the terminal and then sending it. The output is feedback data that can be displayed on the terminal.

[0297] Step 6:

[0298] The device notifies the user of the feedback data it has received. Its input is feedback sent from the server. Its specific actions include playing the feedback message aloud and displaying character animations. As output, the user is provided with visual and auditory feedback.

[0299] Step 7:

[0300] The server anonymizes or deletes processed data to protect users' personal information. The inputs used are the raw data to be analyzed and the intermediate data used to generate feedback. By anonymizing or completely deleting these, the output remains privacy-protected.

[0301] (Application Example 1)

[0302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] In today's commercial environment, improving employee customer service attitudes and behavior is crucial for increasing customer satisfaction and enhancing a store's reputation. However, providing real-time feedback on individual employee etiquette and manners is difficult. To address this challenge, an effective system is needed that allows employees to immediately improve their own customer service attitudes.

[0304] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes means for acquiring the user's voice information and video information, means for analyzing the acquired voice information and video information and evaluating the characteristics related to the user's etiquette and manners, and means for generating a response by a character with selectable displays for the user based on the analysis result. As a result, employees can receive real-time feedback on their attitudes and behaviors during customer service, immediately improve, and enhance customer satisfaction.

[0306] The "user" refers to the person who receives feedback using the system.

[0307] The "voice information" is data related to the voice and sound acquired from the user.

[0308] The "video information" is visual data capturing the user's appearance and actions.

[0309] The "means for acquiring" refers to the devices and methods for electronically capturing voice information and video information.

[0310] The "means for analyzing" is a technology for processing the acquired voice information and video information and extracting the characteristics related to the user's attitude and behavior. [[ID=2Y]]

[0311] "Etiquette and manners" refer to appropriate behaviors and communication styles in line with social norms.

[0312] The "means for evaluating" refers to the mechanism for making judgments by comparing with criteria based on the characteristics obtained from the analyzed data.

[0313] The "character with displays" refers to the character visually or auditorily expressed for conveying feedback to the user.

[0314] The "means for generating a response" refers to the technology for constructing the feedback given to the user based on the evaluation result.

[0315] The system of this invention acquires audio and video information, analyzes it, and provides feedback to the user. Specifically, when an employee wears a device such as smart glasses in a store and interacts with customers, their audio and video are captured in real time. The acquired data is transmitted to a server via a network and analyzed using advanced image recognition and speech recognition algorithms. The server evaluates features related to the user's manners and etiquette and generates feedback based on the results.

[0316] This system processes and analyzes data using software such as OpenCV and Google Speech Recognition. The analysis results are communicated to the user through a character with selectable displays on the user's device. Depending on the user's choice, this character provides feedback in a user-friendly format, either visually or aurally.

[0317] For example, when a new employee at a cafe serves customers for the first time, this system can be used to quickly identify areas for improvement, such as not smiling enough, allowing them to immediately improve their attitude and ultimately increase customer satisfaction.

[0318] An example of a prompt message might be, "Capture the current customer service interaction through your smart glasses and evaluate the frequency of smiles during the interaction. Select a character and provide feedback such as 'Smile more.'" This allows users to understand and implement specific ways to improve their own behavior.

[0319] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0320] Step 1:

[0321] The device uses the smart glasses' camera and microphone to acquire the user's voice and video information in real time. The input to this process is the user's raw voice and video, and the output is digitized voice and video data. This digital data is then transmitted to the next step.

[0322] Step 2:

[0323] The terminal transmits the acquired audio and video information to the server via the network. Data processing during this transmission process includes data compression and format conversion. The output of this step is audio and video data converted into an analyzable format.

[0324] Step 3:

[0325] The server analyzes the received audio and video data using image recognition and speech recognition algorithms. It uses OpenCV to detect facial expressions and gestures from the video data, and Google Speech Recognition to recognize the audio content as text data. The output of this step is data that quantifies or categorizes features related to the user's manners and etiquette.

[0326] Step 4:

[0327] The server generates feedback for the user based on the analysis results. This feedback involves using a generative AI model to synthesize appropriate responses from prompts using a character. The input to this procedure is the analyzed feature data, and the output is text or audio representation as feedback.

[0328] Step 5:

[0329] The server sends the generated feedback to the terminal. The terminal then plays the feedback visually or audibly through a selected character to present it to the user. The output of this process is the feedback information that the user receives directly. This points out attitudes and behaviors that need improvement and includes specific suggestions for improvement.

[0330] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0331] This invention combines an emotion engine with a system that analyzes user voice and video data and provides feedback from selectable characters to improve manners and etiquette. This makes it possible to adjust the feedback according to the user's emotional state.

[0332] First, the user initiates the capture of audio and video data via a dedicated application on their device. The data is sent from the device to a server, which extracts features related to the user's manners and etiquette through image recognition and speech recognition.

[0333] Furthermore, in this invention, an emotion engine embedded in the server analyzes the user's voice tone and facial expressions to recognize emotions in real time. The emotion engine extracts emotional states such as joy, anger, sadness, and surprise, and incorporates this data into the feedback generation process.

[0334] Feedback is delivered in different tones and expressions depending on the character selected by the user. The server adjusts the content and tone of the feedback based on emotional data obtained from the emotion engine. For example, if the server determines that the user is stressed, the feedback will be adjusted to be delivered in gentler language.

[0335] The generated feedback is notified to the user through the device. The notification is presented as a text message, voice message, or as the movement and voice of an animated character. Users can also review the feedback on their device and customize the type and format of the feedback to their preference.

[0336] From the perspective of protecting personal information, the server has the capability to ultimately anonymize or delete data. This allows users to continue receiving feedback in a privacy-protected environment.

[0337] As a concrete example, consider a scenario where a user is practicing conversations at work. The terminal records the user's speech, and the server analyzes the content, tone, and facial expressions of the conversation. The emotion engine detects when the user is feeling nervous and provides relaxing feedback tailored to that emotion. The user receives this feedback and can effectively improve their communication skills. In this way, the system of the present invention enables more advanced self-improvement support than conventional methods.

[0338] The following describes the processing flow.

[0339] Step 1:

[0340] The user launches a dedicated application on their device. Within the application, the user selects options corresponding to the situations in which they want feedback, and, if necessary, sets permissions for camera and microphone use.

[0341] Step 2:

[0342] The device captures the user's audio and video data in real time. The captured data is temporarily stored on the device, and initial processing such as noise reduction and image quality improvement is performed on the data.

[0343] Step 3:

[0344] The terminal transmits processed audio and video data to the server using a secure communication method. During this process, the data is encrypted to prevent unauthorized access by third parties.

[0345] Step 4:

[0346] A machine learning model is applied to analyze the audio and video data received by the server. The server detects characteristics related to the user's manners and etiquette and extracts specific points of criticism.

[0347] Step 5:

[0348] The server uses an emotion engine to analyze the user's emotional state from their voice tone and video. For example, it identifies emotions such as joy, tension, and anger from the user's facial expressions and voice tone.

[0349] Step 6:

[0350] The server uses analysis results and emotional data to generate personalized feedback for the user. The feedback is structured to match the personality of the character selected by the user, with appropriate tone and content. The emphasis and expression of the feedback are adjusted according to the emotional state.

[0351] Step 7:

[0352] The server generates feedback and sends it to the device. The feedback is provided in a format specified by the user, such as synthesized speech, text messages, or animations.

[0353] Step 8:

[0354] The device provides feedback to the user. The user can review the feedback and pay attention to the etiquette improvements recommended by the system. Users can also customize the content and format of the feedback according to their preferences.

[0355] Step 9:

[0356] After the server finishes processing, it anonymizes or deletes user data. Data containing personal information, especially sentiment data, is managed appropriately to ensure privacy. This allows for the provision of continuous feedback while protecting user privacy.

[0357] (Example 2)

[0358] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0359] In modern society, where communication and manners are increasingly important, there is a need for support in helping individuals effectively improve their manners and etiquette. However, traditional methods have struggled to provide appropriate feedback in real time, taking into account individual emotional states. Furthermore, there are insufficient methods to provide personalized feedback while protecting user privacy. To address these challenges, there is a need to develop a system that efficiently and safely improves individuals' communication skills.

[0360] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0361] In this invention, the server includes means for analyzing user information data and detecting features, means for utilizing emotion recognition data for feedback generation, and means for anonymizing or deleting data. This enables advanced self-improvement support tailored to the user's emotional state.

[0362] "Information data" refers to input data, including user voice and video, which the system analyzes to identify features and utilize for feedback generation.

[0363] "Analysis" refers to the process of extracting and identifying user-related characteristics from information data, thereby obtaining data that serves as the basis for feedback.

[0364] "Characteristics" refer to attributes and patterns identified from user information data, representing behaviors and states related to manners and etiquette.

[0365] A "representation model" refers to a selectable character or method of expression used to set the style and tone of the feedback and responses provided to the user.

[0366] A "communication terminal" refers to a device used by a user to receive feedback and notifications generated by the system.

[0367] "Machine learning technology" refers to algorithms and models used in data analysis, which automatically extract features from informational data and contribute to feedback generation.

[0368] "Anonymization" is the process of processing data in a way that makes it impossible to identify personal information, with the aim of protecting privacy.

[0369] This invention is a system that analyzes voice and video data and provides feedback related to manners and etiquette in order to improve the individual communication skills of users. The specific forms in which this system is implemented are described below.

[0370] First, users use a dedicated application on their device to record audio and video data of everyday conversations and specific situations. This data is temporarily stored in random-access memory or storage devices before being sent to a server via a secure communication protocol. The application installed on the device has recording and shooting functions, efficiently collecting user data.

[0371] The server runs image recognition and speech recognition programs to analyze the received data. For example, machine learning libraries such as TensorFlow and PyTorch are used, allowing the server to recognize the user's voice tone and facial expressions. Based on this, an emotion engine is activated to extract the user's emotional state in real time. Emotional states are classified into categories such as joy, anger, sadness, and surprise.

[0372] Furthermore, the server uses this data to generate different feedback depending on the user's chosen representation model (character). The feedback is adjusted according to the user's emotional state; for example, if the server determines that the user is stressed, it will respond using calmer language.

[0373] Users are notified of this generated feedback on their device in visual or auditory form. Notification formats include text messages, voice messages, and the voice of an animated character, and users can choose from these options. Users can also review the feedback and change settings via the application if necessary.

[0374] As an example of how this system works, imagine a scenario where a user is practicing a presentation at work. The user records their speech on their device, and the data is sent to the server. The server analyzes the tone of the conversation and facial expressions to identify when the user is feeling nervous. Then, it generates relaxing feedback and notifies the user on their device.

[0375] A concrete example of a prompt would be, "I would like feedback on how my speaking style and facial expressions are being perceived during presentation practice at work." In this way, users can effectively improve their skills through the system.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The user initiates the capture of audio and video data using a dedicated application. During this process, the user records their speech and facial expressions using the device's camera and microphone. The input is the user's real-time audio and video data, which is captured directly. The output is the captured audio and video data.

[0379] Step 2:

[0380] The terminal sends the captured audio and video data to the server using a secure protocol. SSL / TLS encrypted communication is used for this transmission. The input is the data captured in step 1, which is converted into a format that the server can process and then sent. The output is the completion of data transmission to the server.

[0381] Step 3:

[0382] The server analyzes the received data and uses an image recognition algorithm to analyze the user's facial expressions. In addition, it uses speech recognition software to analyze the tone and intonation of the voice. The input is the audio and video data sent to the server, and the output is the analyzed user characteristic data.

[0383] Step 4:

[0384] The emotion engine embedded in the server identifies the user's emotional state based on the analyzed data. This identification process uses a machine learning model to extract certain emotional categories. The input is the feature data obtained in step 3, and the output is the identified emotional state data.

[0385] Step 5:

[0386] The server uses collected emotional state data to generate personalized feedback based on the user's selected expression model. This feedback includes adjustments to content and tone according to the emotion. The input is emotional state data and the user's selected expression model, and the output is the generated feedback message.

[0387] Step 6:

[0388] The terminal receives feedback sent from the server and notifies the user. Notifications can take the form of text messages, voice messages, or animated visuals. The input is feedback data from the server, and the output is various forms of feedback presented to the user.

[0389] Step 7:

[0390] Users can review the feedback and, if necessary, modify the format and representation model of the feedback through the application. This allows users to explore more effective feedback patterns. The input is the presented feedback and the user's selection, and the output is the new feedback setting adjusted by the user.

[0391] (Application Example 2)

[0392] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0393] In traditional customer service, improving employee manners and service skills is crucial, but there have been limited means of providing objective, real-time feedback. Furthermore, because feedback is often fixed, it's difficult to provide appropriate advice tailored to the individual employee's emotional state. This invention aims to solve these problems and effectively improve employee customer service skills and customer satisfaction.

[0394] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0395] In this invention, the server includes means for receiving user voice and video data, means for an emotion engine that analyzes the user's emotional state and adjusts feedback based on the analysis results, and means for anonymizing or deleting data to protect the user's personal information. This enables the provision of real-time feedback that takes the user's emotions into consideration.

[0396] "User voice data" refers to digital signals containing information spoken by the user, and is the acoustic data that will be analyzed.

[0397] "Video data" refers to digital signals containing information that visually captures the user's movements and facial expressions, and is the visual data that is analyzed.

[0398] "Analysis means" refers to a device or system that processes received audio and video data and executes computational techniques or algorithms to detect features and patterns.

[0399] "Characteristics of manners and etiquette" refer to traits that represent the socially expected norms and politeness regarding a user's behavior and speech.

[0400] A "feedback generation method" is a system that creates information that provides appropriate advice and guidance to the user based on the analyzed data.

[0401] "Emotional state" refers to elements that indicate the user's mental or psychological state, and is inferred from factors such as voice tone and facial expressions.

[0402] An "emotion engine" is a hardware or software module that recognizes emotional states from audio and video data and adjusts appropriate feedback accordingly.

[0403] "Anonymization" is a method of processing data in a way that makes it impossible to identify individuals, and it is a technology for protecting privacy.

[0404] "Means of deletion" refer to the technologies and processes used to erase and make inaccessible stored data.

[0405] This invention is a system primarily aimed at improving manners in customer service at physical stores. The system utilizes smart glasses worn by the user to transmit audio and video data to a server in real time. The smart glasses are equipped with a camera and microphone to capture the user's voice and facial expressions.

[0406] The server uses the Google Cloud Speech-to-Text API to convert audio data into text format and the OpenCV library to analyze facial expressions from video data. This data is then analyzed by an emotion engine using IBM Watson Tone Analyzer to identify emotional states and extract features related to the user's manners and etiquette.

[0407] Based on this, the server uses a Python program to generate feedback appropriate to the selected character. The feedback is displayed in real time on the smart glasses or notified to the user via voice. The tone and content of this feedback are adjusted according to the user's emotional state.

[0408] For example, if it is determined that a store employee is feeling anxious while serving a customer, a message such as "Please relax a little more and smile" will be provided as feedback. Furthermore, from a personal information protection standpoint, the data is anonymized on the server and managed appropriately.

[0409] An example of inputting prompts into a generative AI model is, "Generate feedback to help a store employee relax if they are feeling nervous while serving a customer." This allows the system to effectively improve employees' customer service skills through accurate feedback.

[0410] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0411] Step 1:

[0412] The terminal (smart glasses) captures the user's audio and video data in real time during customer service. The captured data is transmitted to the server via wireless communication. The input for this step is the user's real-time audio and video, and the output is audio and video data in digital format sent to the server.

[0413] Step 2:

[0414] The server analyzes the received audio data using the Google Cloud Speech-to-Text API and converts the audio signal into text data. The input is audio data, and the converted output is text data. The server uses this data to understand what the user is saying.

[0415] Step 3:

[0416] The server analyzes video data using the OpenCV library to analyze the user's facial expressions. The input is video data, and the output is feature data related to the user's facial expressions. This process allows for obtaining real-time information about the user's facial expressions.

[0417] Step 4:

[0418] The server uses IBM Watson Tone Analyzer to identify the emotional state based on the analysis of voice and facial expressions. The input for this step is voice-text data and facial expression feature data, and the output is data indicating the user's emotional state. The server uses the obtained emotional state to adjust the feedback.

[0419] Step 5:

[0420] The server generates feedback using a Python program based on the analyzed emotional state and manners characteristics. The input for this step is emotional state data and manners characteristic data, and the output is the generated feedback message. The feedback is expressed differently depending on the character selected by the user.

[0421] Step 6:

[0422] The device notifies the user of the generated feedback. The notification is displayed on the smart glasses' screen or played as audio. The input for this step is the generated feedback message, and the output is the feedback notification the user receives. This helps the user improve their customer service skills in real time.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] This invention is a system for facilitating feedback on user manners and etiquette, providing feedback to the user through selectable characters via the analysis of audio and video data. The system is primarily configured and operates as follows:

[0440] First, the user launches the application on their device. This application has the ability to capture the user's voice and video in real time using the camera and microphone. The user starts capturing data when they need specific feedback.

[0441] The device transmits captured audio and video data to a server over the network. The server analyzes this data using advanced image and speech recognition algorithms. The analysis identifies features related to user etiquette, such as the detection of nasal hairs or signs of bad breath.

[0442] Based on the results of this analysis, the server generates appropriate feedback through the character selected by the user. This feedback consists of a friendly character's way of speaking, such as, "You have nose hairs sticking out, fix it."

[0443] Next, the generated feedback is sent back to the device and notified to the user visually or audibly. The device uses audio playback and character animation to naturally convey the feedback to the user. This notification is seamless and interactive so as not to interfere with the user's feedback reception.

[0444] Finally, the server strives to protect users' personal information by anonymizing or deleting data. This allows users to continue using the feedback service without worrying about their privacy.

[0445] As a concrete example, imagine a scenario where a young business professional is preparing a presentation. The user launches the app on their device, turns on the camera and microphone, and captures themselves. While the user practices the presentation, the server analyzes feedback on the user's voice clarity and gestures, and points out areas for improvement through an appropriate character. The user receives this feedback immediately and can improve their performance.

[0446] In this way, the system of this invention provides enjoyable and effective support for self-improvement in the user's daily life and business situations.

[0447] The following describes the processing flow.

[0448] Step 1:

[0449] The user launches a dedicated application on their device. The application, with the user's consent, enables the camera and microphone. The user selects the option to start data capture when they want to receive feedback.

[0450] Step 2:

[0451] The device captures the user's audio and video data. This process is performed in real time, and the captured data is temporarily stored on the device. Here, basic processing such as noise reduction is performed to optimize the data quality.

[0452] Step 3:

[0453] The terminal processes the audio and video data and transmits it to the server via the network. The data is encrypted using secure communication technology to protect privacy.

[0454] Step 4:

[0455] The server analyzes the received audio and video data. The machine learning algorithms used here are optimized to detect features related to predefined manners and etiquette. For example, the server analyzes facial images for the presence or absence of nose hairs, and audio for speech clarity.

[0456] Step 5:

[0457] The server generates user-appropriate feedback based on the analysis results. This feedback is tailored to the personality of the character selected by the user. For example, it may generate gentle criticism or humorous comments.

[0458] Step 6:

[0459] The server sends generated feedback to the device. This feedback can be in the form of voice messages, text messages, or animations in which the avatar moves.

[0460] Step 7:

[0461] The device presents feedback from the server to the user. Notifications are provided in the user's preferred visual or auditory form. The device provides feedback to the user in a more understandable way through an interactive user interface.

[0462] Step 8:

[0463] The server protects the user's personal information. After all processing is complete, the server anonymizes the data or, if necessary, completely deletes it. This ensures that users can continue to use the service while maintaining their privacy.

[0464] (Example 1)

[0465] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] Obtaining immediate and effective feedback on etiquette and manners in users' daily lives and work is difficult. Traditional methods require third-party criticism or individual training, which are time-consuming and costly. Furthermore, the feedback is often general and lacks specificity tailored to individual situations. As a result, users miss opportunities to immediately improve their own behavior.

[0467] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0468] In this invention, the server includes means for receiving user voice and video data, means for analyzing the received data to detect features related to the user's behavior, and means for generating feedback to the user through a selected character based on the analysis results. This allows the user to receive specific and personalized feedback on their behavior in real time and take immediate corrective action.

[0469] A "user" refers to an individual who uses the system to receive feedback.

[0470] "Voice data" refers to digital information collected and recorded by a device such as a microphone when a user speaks.

[0471] "Video data" refers to a digital representation of a user's visual information captured by a camera or other device.

[0472] "Analysis" refers to the process of using specific algorithms to identify features related to etiquette and manners in received audio and video data.

[0473] "Features" refer to characteristics related to etiquette and manners extracted from audio and video data, and are indicators used for generating feedback.

[0474] "Feedback" refers to responses or suggestions that provide users with information to improve their behavior, which can help users improve their etiquette and manners.

[0475] A "character" refers to a visual or auditory agent used to convey feedback to the user, providing a sense of familiarity.

[0476] "Anonymization" refers to the process of removing personally identifiable information from data so that users' personal information cannot be identified.

[0477] "Deletion" refers to the process of completely removing data, including a user's personal information, and is one way of protecting privacy.

[0478] "Real-time" refers to a processing method where data acquisition, analysis, and feedback provision are all performed instantly.

[0479] "Interactive" refers to a process in which information and feedback are provided based on interaction with the user, indicating the ability to respond to user reactions.

[0480] This invention describes a specific implementation of a system for providing feedback on user behavior. This system aims to provide feedback tailored to the individual circumstances of the user.

[0481] First, the user launches a dedicated application on their device. The device uses its camera and microphone to capture the user's video and audio data in real time. This data is then transmitted to a server via the network.

[0482] Next, the server analyzes the received data using advanced data analysis algorithms. Here, it uses image recognition and speech recognition technologies powered by machine learning models to identify features related to user behavior. Specifically, this includes detecting nasal hairs and analyzing speech clarity. This process also includes models for facial recognition and speech tone analysis.

[0483] Based on the analysis results, the server generates feedback through the character selected by the user. This feedback uses a friendly tone and animations to provide the user with specific suggestions for improving their behavior.

[0484] The generated feedback is notified to the user by the device. The notification is provided visually through character animations and audibly through audio feedback. This allows the user to receive feedback smoothly and immediately have the opportunity to improve their actions.

[0485] As a concrete example, consider a young business professional preparing a presentation. The user launches the app on their device and turns on the camera and microphone. The server analyzes the user's voice and gestures and provides suggestions for improvement through a friendly character. This feedback allows the user to instantly improve their presentation skills.

[0486] An example of a prompt message might be, "Select a character to point out areas for improvement in your gestures during your business presentation, and generate appropriate feedback." By utilizing such prompt messages, the system can generate feedback tailored to the user's needs.

[0487] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0488] Step 1:

[0489] The user launches a dedicated application on their device. The input is the user's command to launch the application. The output is the application activating the camera and microphone and entering capture mode. The camera begins collecting video data, and the microphone begins collecting audio data.

[0490] Step 2:

[0491] The terminal transmits captured audio and video data to the server in real time. The input is raw data acquired from the terminal's camera and microphone. The specific operation of this step is to compress the data and transmit it over the internet. The output is the server, which receives the encoded data over the network.

[0492] Step 3:

[0493] The server applies an AI model to analyze the received audio and video data. It receives encoded data from the terminal as input. By applying algorithms such as facial recognition and voice tone analysis to this data, it extracts features related to etiquette. The output includes information such as nasal hair detection and pronunciation clarity.

[0494] Step 4:

[0495] The server generates feedback for the user based on the analysis results. It uses the feature information obtained from the analysis as input. It customizes the voice and actions of the selected character to create a friendly feedback message. The output includes the generated feedback message and character animation data.

[0496] Step 5:

[0497] The server sends the generated feedback back to the terminal. It uses the generated feedback data as input. The specific operation involves encoding this data in a format suitable for the terminal and then sending it. The output is feedback data that can be displayed on the terminal.

[0498] Step 6:

[0499] The device notifies the user of the feedback data it has received. Its input is feedback sent from the server. Its specific actions include playing the feedback message aloud and displaying character animations. As output, the user is provided with visual and auditory feedback.

[0500] Step 7:

[0501] The server anonymizes or deletes processed data to protect users' personal information. The inputs used are the raw data to be analyzed and the intermediate data used to generate feedback. By anonymizing or completely deleting these, the output remains privacy-protected.

[0502] (Application Example 1)

[0503] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0504] In today's commercial environment, improving employee customer service attitudes and behavior is crucial for increasing customer satisfaction and enhancing a store's reputation. However, providing real-time feedback on individual employee etiquette and manners is difficult. To address this challenge, an effective system is needed that allows employees to immediately improve their own customer service attitudes.

[0505] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0506] In this invention, the server includes means for acquiring user voice and video information, means for analyzing the acquired voice and video information and evaluating features related to the user's manners and etiquette, and means for generating a response from a character with selectable displays for the user based on the analysis results. This enables employees to receive real-time feedback on their attitude and behavior during customer service, make immediate improvements, and enhance customer satisfaction.

[0507] A "user" refers to a person who uses the system and receives feedback from it.

[0508] "Voice information" refers to data related to the voice and sounds obtained from the user.

[0509] "Visual information" refers to visual data that captures the user's appearance and actions.

[0510] "Means of acquisition" refers to devices and methods for electronically capturing audio and video information.

[0511] "Means of analysis" refers to technologies for processing acquired audio and video information and extracting characteristics related to the user's attitude and behavior.

[0512] "Etiquette and manners" refers to appropriate behavior and communication styles that conform to social norms.

[0513] "Means of evaluation" refers to a mechanism that makes judgments based on criteria derived from the characteristics obtained from the analyzed data.

[0514] A "character with a display" refers to a character that is visually or audibly represented to convey feedback to the user.

[0515] "Means of generating responses" refers to technologies for constructing feedback to be given to users based on evaluation results.

[0516] The system of this invention acquires audio and video information, analyzes it, and provides feedback to the user. Specifically, when an employee wears a device such as smart glasses in a store and interacts with customers, their audio and video are captured in real time. The acquired data is transmitted to a server via a network and analyzed using advanced image recognition and speech recognition algorithms. The server evaluates features related to the user's manners and etiquette and generates feedback based on the results.

[0517] This system processes and analyzes data using software such as OpenCV and Google Speech Recognition. The analysis results are communicated to the user through a character with selectable displays on the user's device. Depending on the user's choice, this character provides feedback in a user-friendly format, either visually or aurally.

[0518] For example, when a new employee at a cafe serves customers for the first time, this system can be used to quickly identify areas for improvement, such as not smiling enough, allowing them to immediately improve their attitude and ultimately increase customer satisfaction.

[0519] An example of a prompt message might be, "Capture the current customer service interaction through your smart glasses and evaluate the frequency of smiles during the interaction. Select a character and provide feedback such as 'Smile more.'" This allows users to understand and implement specific ways to improve their own behavior.

[0520] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0521] Step 1:

[0522] The device uses the smart glasses' camera and microphone to acquire the user's voice and video information in real time. The input to this process is the user's raw voice and video, and the output is digitized voice and video data. This digital data is then transmitted to the next step.

[0523] Step 2:

[0524] The terminal transmits the acquired audio and video information to the server via the network. Data processing during this transmission process includes data compression and format conversion. The output of this step is audio and video data converted into an analyzable format.

[0525] Step 3:

[0526] The server analyzes the received audio and video data using image recognition and speech recognition algorithms. It uses OpenCV to detect facial expressions and gestures from the video data, and Google Speech Recognition to recognize the audio content as text data. The output of this step is data that quantifies or categorizes features related to the user's manners and etiquette.

[0527] Step 4:

[0528] The server generates feedback for the user based on the analysis results. This feedback involves using a generative AI model to synthesize appropriate responses from prompts using a character. The input to this procedure is the analyzed feature data, and the output is text or audio representation as feedback.

[0529] Step 5:

[0530] The server sends the generated feedback to the terminal. The terminal then plays the feedback visually or audibly through a selected character to present it to the user. The output of this process is the feedback information that the user receives directly. This points out attitudes and behaviors that need improvement and includes specific suggestions for improvement.

[0531] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0532] This invention combines an emotion engine with a system that analyzes user voice and video data and provides feedback from selectable characters to improve manners and etiquette. This makes it possible to adjust the feedback according to the user's emotional state.

[0533] First, the user initiates the capture of audio and video data via a dedicated application on their device. The data is sent from the device to a server, which extracts features related to the user's manners and etiquette through image recognition and speech recognition.

[0534] Furthermore, in this invention, an emotion engine embedded in the server analyzes the user's voice tone and facial expressions to recognize emotions in real time. The emotion engine extracts emotional states such as joy, anger, sadness, and surprise, and incorporates this data into the feedback generation process.

[0535] Feedback is delivered in different tones and expressions depending on the character selected by the user. The server adjusts the content and tone of the feedback based on emotional data obtained from the emotion engine. For example, if the server determines that the user is stressed, the feedback will be adjusted to be delivered in gentler language.

[0536] The generated feedback is notified to the user through the device. The notification is presented as a text message, voice message, or as the movement and voice of an animated character. Users can also review the feedback on their device and customize the type and format of the feedback to their preference.

[0537] From the perspective of protecting personal information, the server has the capability to ultimately anonymize or delete data. This allows users to continue receiving feedback in a privacy-protected environment.

[0538] As a concrete example, consider a scenario where a user is practicing conversations at work. The terminal records the user's speech, and the server analyzes the content, tone, and facial expressions of the conversation. The emotion engine detects when the user is feeling nervous and provides relaxing feedback tailored to that emotion. The user receives this feedback and can effectively improve their communication skills. In this way, the system of the present invention enables more advanced self-improvement support than conventional methods.

[0539] The following describes the processing flow.

[0540] Step 1:

[0541] The user launches a dedicated application on their device. Within the application, the user selects options corresponding to the situations in which they want feedback, and, if necessary, sets permissions for camera and microphone use.

[0542] Step 2:

[0543] The device captures the user's audio and video data in real time. The captured data is temporarily stored on the device, and initial processing such as noise reduction and image quality improvement is performed on the data.

[0544] Step 3:

[0545] The terminal transmits processed audio and video data to the server using a secure communication method. During this process, the data is encrypted to prevent unauthorized access by third parties.

[0546] Step 4:

[0547] A machine learning model is applied to analyze the audio and video data received by the server. The server detects characteristics related to the user's manners and etiquette and extracts specific points of criticism.

[0548] Step 5:

[0549] The server uses an emotion engine to analyze the user's emotional state from their voice tone and video. For example, it identifies emotions such as joy, tension, and anger from the user's facial expressions and voice tone.

[0550] Step 6:

[0551] The server uses analysis results and emotional data to generate personalized feedback for the user. The feedback is structured to match the personality of the character selected by the user, with appropriate tone and content. The emphasis and expression of the feedback are adjusted according to the emotional state.

[0552] Step 7:

[0553] The server generates feedback and sends it to the device. The feedback is provided in a format specified by the user, such as synthesized speech, text messages, or animations.

[0554] Step 8:

[0555] The device provides feedback to the user. The user can review the feedback and pay attention to the etiquette improvements recommended by the system. Users can also customize the content and format of the feedback according to their preferences.

[0556] Step 9:

[0557] After the server finishes processing, it anonymizes or deletes user data. Data containing personal information, especially sentiment data, is managed appropriately to ensure privacy. This allows for the provision of continuous feedback while protecting user privacy.

[0558] (Example 2)

[0559] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] In modern society, where communication and manners are increasingly important, there is a need for support in helping individuals effectively improve their manners and etiquette. However, traditional methods have struggled to provide appropriate feedback in real time, taking into account individual emotional states. Furthermore, there are insufficient methods to provide personalized feedback while protecting user privacy. To address these challenges, there is a need to develop a system that efficiently and safely improves individuals' communication skills.

[0561] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0562] In this invention, the server includes means for analyzing user information data and detecting features, means for utilizing emotion recognition data for feedback generation, and means for anonymizing or deleting data. This enables advanced self-improvement support tailored to the user's emotional state.

[0563] "Information data" refers to input data, including user voice and video, which the system analyzes to identify features and utilize for feedback generation.

[0564] "Analysis" refers to the process of extracting and identifying user-related characteristics from information data, thereby obtaining data that serves as the basis for feedback.

[0565] "Characteristics" refer to attributes and patterns identified from user information data, representing behaviors and states related to manners and etiquette.

[0566] A "representation model" refers to a selectable character or method of expression used to set the style and tone of the feedback and responses provided to the user.

[0567] A "communication terminal" refers to a device used by a user to receive feedback and notifications generated by the system.

[0568] "Machine learning technology" refers to algorithms and models used in data analysis, which automatically extract features from informational data and contribute to feedback generation.

[0569] "Anonymization" is the process of processing data in a way that makes it impossible to identify personal information, with the aim of protecting privacy.

[0570] This invention is a system that analyzes voice and video data and provides feedback related to manners and etiquette in order to improve the individual communication skills of users. The specific forms in which this system is implemented are described below.

[0571] First, users use a dedicated application on their device to record audio and video data of everyday conversations and specific situations. This data is temporarily stored in random-access memory or storage devices before being sent to a server via a secure communication protocol. The application installed on the device has recording and shooting functions, efficiently collecting user data.

[0572] The server runs image recognition and speech recognition programs to analyze the received data. For example, machine learning libraries such as TensorFlow and PyTorch are used, allowing the server to recognize the user's voice tone and facial expressions. Based on this, an emotion engine is activated to extract the user's emotional state in real time. Emotional states are classified into categories such as joy, anger, sadness, and surprise.

[0573] Furthermore, the server uses this data to generate different feedback depending on the user's chosen representation model (character). The feedback is adjusted according to the user's emotional state; for example, if the server determines that the user is stressed, it will respond using calmer language.

[0574] Users are notified of this generated feedback on their device in visual or auditory form. Notification formats include text messages, voice messages, and the voice of an animated character, and users can choose from these options. Users can also review the feedback and change settings via the application if necessary.

[0575] As an example of how this system works, imagine a scenario where a user is practicing a presentation at work. The user records their speech on their device, and the data is sent to the server. The server analyzes the tone of the conversation and facial expressions to identify when the user is feeling nervous. Then, it generates relaxing feedback and notifies the user on their device.

[0576] A concrete example of a prompt would be, "I would like feedback on how my speaking style and facial expressions are being perceived during presentation practice at work." In this way, users can effectively improve their skills through the system.

[0577] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0578] Step 1:

[0579] The user initiates the capture of audio and video data using a dedicated application. During this process, the user records their speech and facial expressions using the device's camera and microphone. The input is the user's real-time audio and video data, which is captured directly. The output is the captured audio and video data.

[0580] Step 2:

[0581] The terminal sends the captured audio and video data to the server using a secure protocol. SSL / TLS encrypted communication is used for this transmission. The input is the data captured in step 1, which is converted into a format that the server can process and then sent. The output is the completion of data transmission to the server.

[0582] Step 3:

[0583] The server analyzes the received data and uses an image recognition algorithm to analyze the user's facial expressions. In addition, it uses speech recognition software to analyze the tone and intonation of the voice. The input is the audio and video data sent to the server, and the output is the analyzed user characteristic data.

[0584] Step 4:

[0585] The emotion engine embedded in the server identifies the user's emotional state based on the analyzed data. This identification process uses a machine learning model to extract certain emotional categories. The input is the feature data obtained in step 3, and the output is the identified emotional state data.

[0586] Step 5:

[0587] The server uses collected emotional state data to generate personalized feedback based on the user's selected expression model. This feedback includes adjustments to content and tone according to the emotion. The input is emotional state data and the user's selected expression model, and the output is the generated feedback message.

[0588] Step 6:

[0589] The terminal receives feedback sent from the server and notifies the user. Notifications can take the form of text messages, voice messages, or animated visuals. The input is feedback data from the server, and the output is various forms of feedback presented to the user.

[0590] Step 7:

[0591] Users can review the feedback and, if necessary, modify the format and representation model of the feedback through the application. This allows users to explore more effective feedback patterns. The input is the presented feedback and the user's selection, and the output is the new feedback setting adjusted by the user.

[0592] (Application Example 2)

[0593] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0594] In traditional customer service, improving employee manners and service skills is crucial, but there have been limited means of providing objective, real-time feedback. Furthermore, because feedback is often fixed, it's difficult to provide appropriate advice tailored to the individual employee's emotional state. This invention aims to solve these problems and effectively improve employee customer service skills and customer satisfaction.

[0595] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0596] In this invention, the server includes means for receiving user voice and video data, means for an emotion engine that analyzes the user's emotional state and adjusts feedback based on the analysis results, and means for anonymizing or deleting data to protect the user's personal information. This enables the provision of real-time feedback that takes the user's emotions into consideration.

[0597] "User voice data" refers to digital signals containing information spoken by the user, and is the acoustic data that will be analyzed.

[0598] "Video data" refers to digital signals containing information that visually captures the user's movements and facial expressions, and is the visual data that is analyzed.

[0599] "Analysis means" refers to a device or system that processes received audio and video data and executes computational techniques or algorithms to detect features and patterns.

[0600] "Characteristics of manners and etiquette" refer to traits that represent the socially expected norms and politeness regarding a user's behavior and speech.

[0601] A "feedback generation method" is a system that creates information that provides appropriate advice and guidance to the user based on the analyzed data.

[0602] "Emotional state" refers to elements that indicate the user's mental or psychological state, and is inferred from factors such as voice tone and facial expressions.

[0603] An "emotion engine" is a hardware or software module that recognizes emotional states from audio and video data and adjusts appropriate feedback accordingly.

[0604] "Anonymization" is a method of processing data in a way that makes it impossible to identify individuals, and it is a technology for protecting privacy.

[0605] "Means of deletion" refer to the technologies and processes used to erase and make inaccessible stored data.

[0606] This invention is a system primarily aimed at improving manners in customer service at physical stores. The system utilizes smart glasses worn by the user to transmit audio and video data to a server in real time. The smart glasses are equipped with a camera and microphone to capture the user's voice and facial expressions.

[0607] The server uses the Google Cloud Speech-to-Text API to convert audio data into text format and the OpenCV library to analyze facial expressions from video data. This data is then analyzed by an emotion engine using IBM Watson Tone Analyzer to identify emotional states and extract features related to the user's manners and etiquette.

[0608] Based on this, the server uses a Python program to generate feedback appropriate to the selected character. The feedback is displayed in real time on the smart glasses or notified to the user via voice. The tone and content of this feedback are adjusted according to the user's emotional state.

[0609] For example, if it is determined that a store employee is feeling anxious while serving a customer, a message such as "Please relax a little more and smile" will be provided as feedback. Furthermore, from a personal information protection standpoint, the data is anonymized on the server and managed appropriately.

[0610] An example of inputting prompts into a generative AI model is, "Generate feedback to help a store employee relax if they are feeling nervous while serving a customer." This allows the system to effectively improve employees' customer service skills through accurate feedback.

[0611] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0612] Step 1:

[0613] The terminal (smart glasses) captures the user's audio and video data in real time during customer service. The captured data is transmitted to the server via wireless communication. The input for this step is the user's real-time audio and video, and the output is audio and video data in digital format sent to the server.

[0614] Step 2:

[0615] The server analyzes the received audio data using the Google Cloud Speech-to-Text API and converts the audio signal into text data. The input is audio data, and the converted output is text data. The server uses this data to understand what the user is saying.

[0616] Step 3:

[0617] The server analyzes video data using the OpenCV library to analyze the user's facial expressions. The input is video data, and the output is feature data related to the user's facial expressions. This process allows for obtaining real-time information about the user's facial expressions.

[0618] Step 4:

[0619] The server uses IBM Watson Tone Analyzer to identify the emotional state based on the analysis of voice and facial expressions. The input for this step is voice-text data and facial expression feature data, and the output is data indicating the user's emotional state. The server uses the obtained emotional state to adjust the feedback.

[0620] Step 5:

[0621] The server generates feedback using a Python program based on the analyzed emotional state and manners characteristics. The input for this step is emotional state data and manners characteristic data, and the output is the generated feedback message. The feedback is expressed differently depending on the character selected by the user.

[0622] Step 6:

[0623] The device notifies the user of the generated feedback. The notification is displayed on the smart glasses' screen or played as audio. The input for this step is the generated feedback message, and the output is the feedback notification the user receives. This helps the user improve their customer service skills in real time.

[0624] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0625] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0627] [Fourth Embodiment]

[0628] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0629] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0630] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0632] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0634] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0635] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0636] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0637] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0640] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] This invention is a system for facilitating feedback on user manners and etiquette, providing feedback to the user through selectable characters via the analysis of audio and video data. The system is primarily configured and operates as follows:

[0642] First, the user launches the application on their device. This application has the ability to capture the user's voice and video in real time using the camera and microphone. The user starts capturing data when they need specific feedback.

[0643] The device transmits captured audio and video data to a server over the network. The server analyzes this data using advanced image and speech recognition algorithms. The analysis identifies features related to user etiquette, such as the detection of nasal hairs or signs of bad breath.

[0644] Based on the results of this analysis, the server generates appropriate feedback through the character selected by the user. This feedback consists of a friendly character's way of speaking, such as, "You have nose hairs sticking out, fix it."

[0645] Next, the generated feedback is sent back to the device and notified to the user visually or audibly. The device uses audio playback and character animation to naturally convey the feedback to the user. This notification is seamless and interactive so as not to interfere with the user's feedback reception.

[0646] Finally, the server strives to protect users' personal information by anonymizing or deleting data. This allows users to continue using the feedback service without worrying about their privacy.

[0647] As a concrete example, imagine a scenario where a young business professional is preparing a presentation. The user launches the app on their device, turns on the camera and microphone, and captures themselves. While the user practices the presentation, the server analyzes feedback on the user's voice clarity and gestures, and points out areas for improvement through an appropriate character. The user receives this feedback immediately and can improve their performance.

[0648] In this way, the system of this invention provides enjoyable and effective support for self-improvement in the user's daily life and business situations.

[0649] The following describes the processing flow.

[0650] Step 1:

[0651] The user launches a dedicated application on their device. The application, with the user's consent, enables the camera and microphone. The user selects the option to start data capture when they want to receive feedback.

[0652] Step 2:

[0653] The device captures the user's audio and video data. This process is performed in real time, and the captured data is temporarily stored on the device. Here, basic processing such as noise reduction is performed to optimize the data quality.

[0654] Step 3:

[0655] The terminal processes the audio and video data and transmits it to the server via the network. The data is encrypted using secure communication technology to protect privacy.

[0656] Step 4:

[0657] The server analyzes the received audio and video data. The machine learning algorithms used here are optimized to detect features related to predefined manners and etiquette. For example, the server analyzes facial images for the presence or absence of nose hairs, and audio for speech clarity.

[0658] Step 5:

[0659] The server generates user-appropriate feedback based on the analysis results. This feedback is tailored to the personality of the character selected by the user. For example, it may generate gentle criticism or humorous comments.

[0660] Step 6:

[0661] The server sends generated feedback to the device. This feedback can be in the form of voice messages, text messages, or animations in which the avatar moves.

[0662] Step 7:

[0663] The device presents feedback from the server to the user. Notifications are provided in the user's preferred visual or auditory form. The device provides feedback to the user in a more understandable way through an interactive user interface.

[0664] Step 8:

[0665] The server protects the user's personal information. After all processing is complete, the server anonymizes the data or, if necessary, completely deletes it. This ensures that users can continue to use the service while maintaining their privacy.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] Obtaining immediate and effective feedback on etiquette and manners in users' daily lives and work is difficult. Traditional methods require third-party criticism or individual training, which are time-consuming and costly. Furthermore, the feedback is often general and lacks specificity tailored to individual situations. As a result, users miss opportunities to immediately improve their own behavior.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes means for receiving user voice and video data, means for analyzing the received data to detect features related to the user's behavior, and means for generating feedback to the user through a selected character based on the analysis results. This allows the user to receive specific and personalized feedback on their behavior in real time and take immediate corrective action.

[0671] A "user" refers to an individual who uses the system to receive feedback.

[0672] "Voice data" refers to digital information collected and recorded by a device such as a microphone when a user speaks.

[0673] "Video data" refers to a digital representation of a user's visual information captured by a camera or other device.

[0674] "Analysis" refers to the process of using specific algorithms to identify features related to etiquette and manners in received audio and video data.

[0675] "Features" refer to characteristics related to etiquette and manners extracted from audio and video data, and are indicators used for generating feedback.

[0676] "Feedback" refers to responses or suggestions that provide users with information to improve their behavior, which can help users improve their etiquette and manners.

[0677] A "character" refers to a visual or auditory agent used to convey feedback to the user, providing a sense of familiarity.

[0678] "Anonymization" refers to the process of removing personally identifiable information from data so that users' personal information cannot be identified.

[0679] "Deletion" refers to the process of completely removing data, including a user's personal information, and is one way of protecting privacy.

[0680] "Real-time" refers to a processing method where data acquisition, analysis, and feedback provision are all performed instantly.

[0681] "Interactive" refers to a process in which information and feedback are provided based on interaction with the user, indicating the ability to respond to user reactions.

[0682] This invention describes a specific implementation of a system for providing feedback on user behavior. This system aims to provide feedback tailored to the individual circumstances of the user.

[0683] First, the user launches a dedicated application on their device. The device uses its camera and microphone to capture the user's video and audio data in real time. This data is then transmitted to a server via the network.

[0684] Next, the server analyzes the received data using advanced data analysis algorithms. Here, it uses image recognition and speech recognition technologies powered by machine learning models to identify features related to user behavior. Specifically, this includes detecting nasal hairs and analyzing speech clarity. This process also includes models for facial recognition and speech tone analysis.

[0685] Based on the analysis results, the server generates feedback through the character selected by the user. This feedback uses a friendly tone and animations to provide the user with specific suggestions for improving their behavior.

[0686] The generated feedback is notified to the user by the device. The notification is provided visually through character animations and audibly through audio feedback. This allows the user to receive feedback smoothly and immediately have the opportunity to improve their actions.

[0687] As a concrete example, consider a young business professional preparing a presentation. The user launches the app on their device and turns on the camera and microphone. The server analyzes the user's voice and gestures and provides suggestions for improvement through a friendly character. This feedback allows the user to instantly improve their presentation skills.

[0688] An example of a prompt message might be, "Select a character to point out areas for improvement in your gestures during your business presentation, and generate appropriate feedback." By utilizing such prompt messages, the system can generate feedback tailored to the user's needs.

[0689] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0690] Step 1:

[0691] The user launches a dedicated application on their device. The input is the user's command to launch the application. The output is the application activating the camera and microphone and entering capture mode. The camera begins collecting video data, and the microphone begins collecting audio data.

[0692] Step 2:

[0693] The terminal transmits captured audio and video data to the server in real time. The input is raw data acquired from the terminal's camera and microphone. The specific operation of this step is to compress the data and transmit it over the internet. The output is the server, which receives the encoded data over the network.

[0694] Step 3:

[0695] The server applies an AI model to analyze the received audio and video data. It receives encoded data from the terminal as input. By applying algorithms such as facial recognition and voice tone analysis to this data, it extracts features related to etiquette. The output includes information such as nasal hair detection and pronunciation clarity.

[0696] Step 4:

[0697] The server generates feedback for the user based on the analysis results. It uses the feature information obtained from the analysis as input. It customizes the voice and actions of the selected character to create a friendly feedback message. The output includes the generated feedback message and character animation data.

[0698] Step 5:

[0699] The server sends the generated feedback back to the terminal. It uses the generated feedback data as input. The specific operation involves encoding this data in a format suitable for the terminal and then sending it. The output is feedback data that can be displayed on the terminal.

[0700] Step 6:

[0701] The device notifies the user of the feedback data it has received. Its input is feedback sent from the server. Its specific actions include playing the feedback message aloud and displaying character animations. As output, the user is provided with visual and auditory feedback.

[0702] Step 7:

[0703] The server anonymizes or deletes processed data to protect users' personal information. The inputs used are the raw data to be analyzed and the intermediate data used to generate feedback. By anonymizing or completely deleting these, the output remains privacy-protected.

[0704] (Application Example 1)

[0705] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] In today's commercial environment, improving employee customer service attitudes and behavior is crucial for increasing customer satisfaction and enhancing a store's reputation. However, providing real-time feedback on individual employee etiquette and manners is difficult. To address this challenge, an effective system is needed that allows employees to immediately improve their own customer service attitudes.

[0707] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0708] In this invention, the server includes means for acquiring user voice and video information, means for analyzing the acquired voice and video information and evaluating features related to the user's manners and etiquette, and means for generating a response from a character with selectable displays for the user based on the analysis results. This enables employees to receive real-time feedback on their attitude and behavior during customer service, make immediate improvements, and enhance customer satisfaction.

[0709] A "user" refers to a person who uses the system and receives feedback from it.

[0710] "Voice information" refers to data related to the voice and sounds obtained from the user.

[0711] "Visual information" refers to visual data that captures the user's appearance and actions.

[0712] "Means of acquisition" refers to devices and methods for electronically capturing audio and video information.

[0713] "Means of analysis" refers to technologies for processing acquired audio and video information and extracting characteristics related to the user's attitude and behavior.

[0714] "Etiquette and manners" refers to appropriate behavior and communication styles that conform to social norms.

[0715] "Means of evaluation" refers to a mechanism that makes judgments based on criteria derived from the characteristics obtained from the analyzed data.

[0716] A "character with a display" refers to a character that is visually or audibly represented to convey feedback to the user.

[0717] "Means of generating responses" refers to technologies for constructing feedback to be given to users based on evaluation results.

[0718] The system of this invention acquires audio and video information, analyzes it, and provides feedback to the user. Specifically, when an employee wears a device such as smart glasses in a store and interacts with customers, their audio and video are captured in real time. The acquired data is transmitted to a server via a network and analyzed using advanced image recognition and speech recognition algorithms. The server evaluates features related to the user's manners and etiquette and generates feedback based on the results.

[0719] This system processes and analyzes data using software such as OpenCV and Google Speech Recognition. The analysis results are communicated to the user through a character with selectable displays on the user's device. Depending on the user's choice, this character provides feedback in a user-friendly format, either visually or aurally.

[0720] For example, when a new employee at a cafe serves customers for the first time, this system can be used to quickly identify areas for improvement, such as not smiling enough, allowing them to immediately improve their attitude and ultimately increase customer satisfaction.

[0721] An example of a prompt message might be, "Capture the current customer service interaction through your smart glasses and evaluate the frequency of smiles during the interaction. Select a character and provide feedback such as 'Smile more.'" This allows users to understand and implement specific ways to improve their own behavior.

[0722] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0723] Step 1:

[0724] The device uses the smart glasses' camera and microphone to acquire the user's voice and video information in real time. The input to this process is the user's raw voice and video, and the output is digitized voice and video data. This digital data is then transmitted to the next step.

[0725] Step 2:

[0726] The terminal transmits the acquired audio and video information to the server via the network. Data processing during this transmission process includes data compression and format conversion. The output of this step is audio and video data converted into an analyzable format.

[0727] Step 3:

[0728] The server analyzes the received audio and video data using image recognition and speech recognition algorithms. It uses OpenCV to detect facial expressions and gestures from the video data, and Google Speech Recognition to recognize the audio content as text data. The output of this step is data that quantifies or categorizes features related to the user's manners and etiquette.

[0729] Step 4:

[0730] The server generates feedback for the user based on the analysis results. This feedback involves using a generative AI model to synthesize appropriate responses from prompts using a character. The input to this procedure is the analyzed feature data, and the output is text or audio representation as feedback.

[0731] Step 5:

[0732] The server sends the generated feedback to the terminal. The terminal then plays the feedback visually or audibly through a selected character to present it to the user. The output of this process is the feedback information that the user receives directly. This points out attitudes and behaviors that need improvement and includes specific suggestions for improvement.

[0733] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0734] This invention combines an emotion engine with a system that analyzes user voice and video data and provides feedback from selectable characters to improve manners and etiquette. This makes it possible to adjust the feedback according to the user's emotional state.

[0735] First, the user initiates the capture of audio and video data via a dedicated application on their device. The data is sent from the device to a server, which extracts features related to the user's manners and etiquette through image recognition and speech recognition.

[0736] Furthermore, in this invention, an emotion engine embedded in the server analyzes the user's voice tone and facial expressions to recognize emotions in real time. The emotion engine extracts emotional states such as joy, anger, sadness, and surprise, and incorporates this data into the feedback generation process.

[0737] Feedback is delivered in different tones and expressions depending on the character selected by the user. The server adjusts the content and tone of the feedback based on emotional data obtained from the emotion engine. For example, if the server determines that the user is stressed, the feedback will be adjusted to be delivered in gentler language.

[0738] The generated feedback is notified to the user through the device. The notification is presented as a text message, voice message, or as the movement and voice of an animated character. Users can also review the feedback on their device and customize the type and format of the feedback to their preference.

[0739] From the perspective of protecting personal information, the server has the capability to ultimately anonymize or delete data. This allows users to continue receiving feedback in a privacy-protected environment.

[0740] As a concrete example, consider a scenario where a user is practicing conversations at work. The terminal records the user's speech, and the server analyzes the content, tone, and facial expressions of the conversation. The emotion engine detects when the user is feeling nervous and provides relaxing feedback tailored to that emotion. The user receives this feedback and can effectively improve their communication skills. In this way, the system of the present invention enables more advanced self-improvement support than conventional methods.

[0741] The following describes the processing flow.

[0742] Step 1:

[0743] The user launches a dedicated application on their device. Within the application, the user selects options corresponding to the situations in which they want feedback, and, if necessary, sets permissions for camera and microphone use.

[0744] Step 2:

[0745] The device captures the user's audio and video data in real time. The captured data is temporarily stored on the device, and initial processing such as noise reduction and image quality improvement is performed on the data.

[0746] Step 3:

[0747] The terminal transmits processed audio and video data to the server using a secure communication method. During this process, the data is encrypted to prevent unauthorized access by third parties.

[0748] Step 4:

[0749] A machine learning model is applied to analyze the audio and video data received by the server. The server detects characteristics related to the user's manners and etiquette and extracts specific points of criticism.

[0750] Step 5:

[0751] The server uses an emotion engine to analyze the user's emotional state from their voice tone and video. For example, it identifies emotions such as joy, tension, and anger from the user's facial expressions and voice tone.

[0752] Step 6:

[0753] The server uses analysis results and emotional data to generate personalized feedback for the user. The feedback is structured to match the personality of the character selected by the user, with appropriate tone and content. The emphasis and expression of the feedback are adjusted according to the emotional state.

[0754] Step 7:

[0755] The server generates feedback and sends it to the device. The feedback is provided in a format specified by the user, such as synthesized speech, text messages, or animations.

[0756] Step 8:

[0757] The device provides feedback to the user. The user can review the feedback and pay attention to the etiquette improvements recommended by the system. Users can also customize the content and format of the feedback according to their preferences.

[0758] Step 9:

[0759] After the server finishes processing, it anonymizes or deletes user data. Data containing personal information, especially sentiment data, is managed appropriately to ensure privacy. This allows for the provision of continuous feedback while protecting user privacy.

[0760] (Example 2)

[0761] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] In modern society, where communication and manners are increasingly important, there is a need for support in helping individuals effectively improve their manners and etiquette. However, traditional methods have struggled to provide appropriate feedback in real time, taking into account individual emotional states. Furthermore, there are insufficient methods to provide personalized feedback while protecting user privacy. To address these challenges, there is a need to develop a system that efficiently and safely improves individuals' communication skills.

[0763] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0764] In this invention, the server includes means for analyzing user information data and detecting features, means for utilizing emotion recognition data for feedback generation, and means for anonymizing or deleting data. This enables advanced self-improvement support tailored to the user's emotional state.

[0765] "Information data" refers to input data, including user voice and video, which the system analyzes to identify features and utilize for feedback generation.

[0766] "Analysis" refers to the process of extracting and identifying user-related characteristics from information data, thereby obtaining data that serves as the basis for feedback.

[0767] "Characteristics" refer to attributes and patterns identified from user information data, representing behaviors and states related to manners and etiquette.

[0768] A "representation model" refers to a selectable character or method of expression used to set the style and tone of the feedback and responses provided to the user.

[0769] A "communication terminal" refers to a device used by a user to receive feedback and notifications generated by the system.

[0770] "Machine learning technology" refers to algorithms and models used in data analysis, which automatically extract features from informational data and contribute to feedback generation.

[0771] "Anonymization" is the process of processing data in a way that makes it impossible to identify personal information, with the aim of protecting privacy.

[0772] This invention is a system that analyzes voice and video data and provides feedback related to manners and etiquette in order to improve the individual communication skills of users. The specific forms in which this system is implemented are described below.

[0773] First, users use a dedicated application on their device to record audio and video data of everyday conversations and specific situations. This data is temporarily stored in random-access memory or storage devices before being sent to a server via a secure communication protocol. The application installed on the device has recording and shooting functions, efficiently collecting user data.

[0774] The server runs image recognition and speech recognition programs to analyze the received data. For example, machine learning libraries such as TensorFlow and PyTorch are used, allowing the server to recognize the user's voice tone and facial expressions. Based on this, an emotion engine is activated to extract the user's emotional state in real time. Emotional states are classified into categories such as joy, anger, sadness, and surprise.

[0775] Furthermore, the server uses this data to generate different feedback depending on the user's chosen representation model (character). The feedback is adjusted according to the user's emotional state; for example, if the server determines that the user is stressed, it will respond using calmer language.

[0776] Users are notified of this generated feedback on their device in visual or auditory form. Notification formats include text messages, voice messages, and the voice of an animated character, and users can choose from these options. Users can also review the feedback and change settings via the application if necessary.

[0777] As an example of how this system works, imagine a scenario where a user is practicing a presentation at work. The user records their speech on their device, and the data is sent to the server. The server analyzes the tone of the conversation and facial expressions to identify when the user is feeling nervous. Then, it generates relaxing feedback and notifies the user on their device.

[0778] A concrete example of a prompt would be, "I would like feedback on how my speaking style and facial expressions are being perceived during presentation practice at work." In this way, users can effectively improve their skills through the system.

[0779] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0780] Step 1:

[0781] The user initiates the capture of audio and video data using a dedicated application. During this process, the user records their speech and facial expressions using the device's camera and microphone. The input is the user's real-time audio and video data, which is captured directly. The output is the captured audio and video data.

[0782] Step 2:

[0783] The terminal sends the captured audio and video data to the server using a secure protocol. SSL / TLS encrypted communication is used for this transmission. The input is the data captured in step 1, which is converted into a format that the server can process and then sent. The output is the completion of data transmission to the server.

[0784] Step 3:

[0785] The server analyzes the received data and uses an image recognition algorithm to analyze the user's facial expressions. In addition, it uses speech recognition software to analyze the tone and intonation of the voice. The input is the audio and video data sent to the server, and the output is the analyzed user characteristic data.

[0786] Step 4:

[0787] The emotion engine embedded in the server identifies the user's emotional state based on the analyzed data. This identification process uses a machine learning model to extract certain emotional categories. The input is the feature data obtained in step 3, and the output is the identified emotional state data.

[0788] Step 5:

[0789] The server uses collected emotional state data to generate personalized feedback based on the user's selected expression model. This feedback includes adjustments to content and tone according to the emotion. The input is emotional state data and the user's selected expression model, and the output is the generated feedback message.

[0790] Step 6:

[0791] The terminal receives feedback sent from the server and notifies the user. Notifications can take the form of text messages, voice messages, or animated visuals. The input is feedback data from the server, and the output is various forms of feedback presented to the user.

[0792] Step 7:

[0793] Users can review the feedback and, if necessary, modify the format and representation model of the feedback through the application. This allows users to explore more effective feedback patterns. The input is the presented feedback and the user's selection, and the output is the new feedback setting adjusted by the user.

[0794] (Application Example 2)

[0795] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0796] In traditional customer service, improving employee manners and service skills is crucial, but there have been limited means of providing objective, real-time feedback. Furthermore, because feedback is often fixed, it's difficult to provide appropriate advice tailored to the individual employee's emotional state. This invention aims to solve these problems and effectively improve employee customer service skills and customer satisfaction.

[0797] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0798] In this invention, the server includes means for receiving user voice and video data, means for an emotion engine that analyzes the user's emotional state and adjusts feedback based on the analysis results, and means for anonymizing or deleting data to protect the user's personal information. This enables the provision of real-time feedback that takes the user's emotions into consideration.

[0799] "User voice data" refers to digital signals containing information spoken by the user, and is the acoustic data that will be analyzed.

[0800] "Video data" refers to digital signals containing information that visually captures the user's movements and facial expressions, and is the visual data that is analyzed.

[0801] "Analysis means" refers to a device or system that processes received audio and video data and executes computational techniques or algorithms to detect features and patterns.

[0802] "Characteristics of manners and etiquette" refer to traits that represent the socially expected norms and politeness regarding a user's behavior and speech.

[0803] A "feedback generation method" is a system that creates information that provides appropriate advice and guidance to the user based on the analyzed data.

[0804] "Emotional state" refers to elements that indicate the user's mental or psychological state, and is inferred from factors such as voice tone and facial expressions.

[0805] An "emotion engine" is a hardware or software module that recognizes emotional states from audio and video data and adjusts appropriate feedback accordingly.

[0806] "Anonymization" is a method of processing data in a way that makes it impossible to identify individuals, and it is a technology for protecting privacy.

[0807] "Means of deletion" refer to the technologies and processes used to erase and make inaccessible stored data.

[0808] This invention is a system primarily aimed at improving manners in customer service at physical stores. The system utilizes smart glasses worn by the user to transmit audio and video data to a server in real time. The smart glasses are equipped with a camera and microphone to capture the user's voice and facial expressions.

[0809] The server uses the Google Cloud Speech-to-Text API to convert audio data into text format and the OpenCV library to analyze facial expressions from video data. This data is then analyzed by an emotion engine using IBM Watson Tone Analyzer to identify emotional states and extract features related to the user's manners and etiquette.

[0810] Based on this, the server uses a Python program to generate feedback appropriate to the selected character. The feedback is displayed in real time on the smart glasses or notified to the user via voice. The tone and content of this feedback are adjusted according to the user's emotional state.

[0811] For example, if it is determined that a store employee is feeling anxious while serving a customer, a message such as "Please relax a little more and smile" will be provided as feedback. Furthermore, from a personal information protection standpoint, the data is anonymized on the server and managed appropriately.

[0812] An example of inputting prompts into a generative AI model is, "Generate feedback to help a store employee relax if they are feeling nervous while serving a customer." This allows the system to effectively improve employees' customer service skills through accurate feedback.

[0813] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0814] Step 1:

[0815] The terminal (smart glasses) captures the user's audio and video data in real time during customer service. The captured data is transmitted to the server via wireless communication. The input for this step is the user's real-time audio and video, and the output is audio and video data in digital format sent to the server.

[0816] Step 2:

[0817] The server analyzes the received audio data using the Google Cloud Speech-to-Text API and converts the audio signal into text data. The input is audio data, and the converted output is text data. The server uses this data to understand what the user is saying.

[0818] Step 3:

[0819] The server analyzes video data using the OpenCV library to analyze the user's facial expressions. The input is video data, and the output is feature data related to the user's facial expressions. This process allows for obtaining real-time information about the user's facial expressions.

[0820] Step 4:

[0821] The server uses IBM Watson Tone Analyzer to identify the emotional state based on the analysis of voice and facial expressions. The input for this step is voice-text data and facial expression feature data, and the output is data indicating the user's emotional state. The server uses the obtained emotional state to adjust the feedback.

[0822] Step 5:

[0823] The server generates feedback using a Python program based on the analyzed emotional state and manners characteristics. The input for this step is emotional state data and manners characteristic data, and the output is the generated feedback message. The feedback is expressed differently depending on the character selected by the user.

[0824] Step 6:

[0825] The device notifies the user of the generated feedback. The notification is displayed on the smart glasses' screen or played as audio. The input for this step is the generated feedback message, and the output is the feedback notification the user receives. This helps the user improve their customer service skills in real time.

[0826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0827] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0829] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0830] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0831] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0832] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0833] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0834] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0836] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0837] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0838] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0840] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0841] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0842] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0843] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0844] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0845] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0846] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0847] The following is further disclosed regarding the embodiments described above.

[0848] (Claim 1)

[0849] Means for receiving user audio and video data,

[0850] A means for analyzing received audio and video data to detect features related to user manners and etiquette,

[0851] A means for generating user-selectable character feedback based on analysis results,

[0852] A means of notifying the user's device of the generated feedback,

[0853] Means to anonymize or delete data in order to protect users' personal information,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, wherein the feedback generation means provides feedback in a visual or auditory form according to the user's choice.

[0857] (Claim 3)

[0858] The system according to claim 1, wherein the data analysis means automatically extracts specific features from the user's voice or video using a machine learning model and uses them to generate feedback.

[0859] "Example 1"

[0860] (Claim 1)

[0861] Means for receiving user audio and video data,

[0862] A means for analyzing received audio and video data and detecting features related to user behavior,

[0863] A means for generating interactive character feedback for the user based on the analysis results,

[0864] A means for notifying the user's information display device of the generated feedback,

[0865] Means to anonymize or delete data in order to protect users' personal information,

[0866] A means for analyzing audio and video data sequentially in real time,

[0867] Means of providing feedback visually and audibly through character images and sounds,

[0868] A system that includes this.

[0869] (Claim 2)

[0870] The system according to claim 1, wherein the feedback generation means provides feedback in a visual or auditory form according to the user's choice and provides interactive notification.

[0871] (Claim 3)

[0872] The system according to claim 1, wherein the data analysis means automatically extracts specific features from the user's voice or video using a generative artificial intelligence model and uses them to generate feedback.

[0873] "Application Example 1"

[0874] (Claim 1)

[0875] Means for acquiring user audio and video information,

[0876] A means for analyzing acquired audio and video information and evaluating features related to the user's manners and etiquette,

[0877] A means for generating a response from a character with selectable displays for the user based on the analysis results,

[0878] Means for displaying the generated response to the user's device,

[0879] Means to anonymize or delete information in order to protect users' personal information,

[0880] A means of providing feedback on the attitude and behavior of store staff during customer interactions in order to improve customer service,

[0881] A device that includes this.

[0882] (Claim 2)

[0883] The apparatus according to claim 1, wherein the response generating means displays the response in a visual or auditory form according to the user's selection.

[0884] (Claim 3)

[0885] The apparatus according to claim 1, wherein the information analysis means automatically extracts specific features from the user's voice or video using a machine learning algorithm and uses them to generate a response.

[0886] "Example 2 of combining an emotion engine"

[0887] (Claim 1)

[0888] Means for receiving user information data,

[0889] A means for analyzing received information data and detecting characteristics related to the user's manners and etiquette,

[0890] A means for generating a response using a selectable representation model based on the analysis results and emotion recognition data,

[0891] A means of notifying the user's communication terminal of the generated response,

[0892] Means to anonymize or delete data in order to protect users' personal information,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, wherein the response generating means provides a response in visual or auditory form according to the user's choice.

[0896] (Claim 3)

[0897] The system according to claim 1, wherein the information analysis means automatically extracts specific features from user information using machine learning technology and uses them to generate a response.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] Means for receiving user audio and video data,

[0901] A means for analyzing received audio and video data to detect features related to user manners and etiquette,

[0902] A means for generating user-selectable character feedback based on analysis results,

[0903] A means of notifying the user's device of the generated feedback,

[0904] A means comprising an emotion engine that analyzes the user's emotional state and adjusts feedback based on the analysis results,

[0905] Means to anonymize or delete data in order to protect users' personal information,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, wherein the feedback generation means provides feedback in visual or auditory form according to the user's choice and is adjusted in real time to match the user's emotional state.

[0909] (Claim 3)

[0910] The system according to claim 1, wherein the data analysis means automatically extracts specific features from the user's voice or video using a machine learning model, and combines the extracted features with the emotional state to generate feedback. [Explanation of Symbols]

[0911] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for receiving user audio and video data, A means for analyzing received audio and video data to detect features related to user manners and etiquette, A means for generating user-selectable character feedback based on analysis results, A means of notifying the user's device of the generated feedback, Means to anonymize or delete data in order to protect users' personal information, A system that includes this.

2. The system according to claim 1, wherein the feedback generation means provides feedback in a visual or auditory form according to the user's choice.

3. The system according to claim 1, wherein the data analysis means automatically extracts specific features from the user's voice or video using a machine learning model and uses them to generate feedback.

Images