system

Abstract

Provide a system. [Solution] a means for constructing a database for storing data relating to sales techniques; A means for recording and recording the user's sales performance; A means for transmitting video and audio data to a server; The server analyzes the video and audio data to evaluate sales techniques, facial expressions, and tone of voice. means for generating feedback based on the analysis results; means for transmitting feedback to a user terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, efficient learning of sales techniques and standardization of skills are important issues. However, traditional sales training methods often involve face-to-face instruction, which is limited by time and location, resulting in problems with the efficiency of new employees' rapid start-up and self-improvement. Furthermore, individual sales skills tend to vary, making it difficult to ensure uniform sales capabilities. [Means for solving the problem]

[0005] The present invention provides a system for promoting efficient learning of sales techniques and skill leveling. This system includes the following means: First, it includes a means for constructing a database for storing data related to sales techniques. Second, it includes a means for videotaping and recording a user's sales performance. It also includes a means for transmitting the videotaped and recorded data to a server. The server includes a means for analyzing the videotaped and recorded data and evaluating the sales technique, facial expression, and tone of voice. It also includes a means for generating feedback based on the analysis results. Finally, it includes a means for transmitting the feedback to the user's terminal. This allows the user to view the feedback and reattempt their sales performance based on the areas for improvement, thereby encouraging self-improvement. Furthermore, it is possible to use a deep learning model to analyze the videotaped and recorded data, enabling highly accurate analysis and feedback.

[0006] A "database" is a device for storing, managing, and searching various data related to sales techniques.

[0007] "Video and audio recording means" means a device or function that records the user's sales performance as video and audio.

[0008] A "server" is a computer system that stores, analyzes, and transmits data over a network.

[0009] "Analysis means" refers to an algorithm or model that analyzes video and audio data and evaluates sales techniques, facial expressions, and tone of voice.

[0010] "Feedback" refers to information for evaluation and improvement provided to the user based on the results of the analysis.

[0011] A "deep learning model" is a machine learning model that uses neural networks to perform complex pattern recognition and data analysis.

[0012] A "user terminal" is a device used by a user, such as a computer, smartphone, or tablet. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system for efficiently learning sales techniques and leveling skills, and is implemented by the following procedure and configuration.

[0035] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0036] Next, users can use the device to record and record their sales performance. Users can use the camera and microphone to record presentations and conversations with customers, allowing them to review their sales techniques later.

[0037] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0038] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[0039] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0040] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0041] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, the user will receive feedback such as "You should smile more." If the tone of voice is appropriate, the user will be evaluated as "Very good tone of voice." This allows the user to clearly understand their strengths and areas for improvement and use this information in their next performance.

[0042] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and provides a powerful tool for users to improve themselves.

[0043] The processing flow will be explained below.

[0044] Step 1:

[0045] The server creates a database of sales techniques. Detailed information such as model sales scenarios, facial expressions, tone and volume of voice is entered into this database. For example, scenarios such as "initial greeting with a customer," "progression of a negotiation," and "successful closing" are saved.

[0046] Step 2:

[0047] Users can use their devices to record and record their sales performance. Using a camera and microphone, users can record their speech and facial expressions for later analysis. For example, they can conduct a mock sales meeting in front of the camera.

[0048] Step 3:

[0049] The device sends the recorded performance data to a server, where it is securely uploaded and stored. This is done using the data upload function via the internet.

[0050] Step 4:

[0051] The server analyzes the received data, using highly accurate analytical techniques such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling face."

[0052] Step 5:

[0053] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique and specific points for improvement such as facial expressions and tone of voice. For example, the server may provide advice such as, "Your tone of voice is good, but you should smile a bit more."

[0054] Step 6:

[0055] The server sends the generated feedback to the terminal, where it is displayed for the user to view. The feedback is provided in a format that is easy for the user to view.

[0056] Step 7:

[0057] Users check the feedback on their device. Based on the feedback provided, they can understand the strengths and areas for improvement in their sales performance. For example, if they are evaluated as having an "appropriate tone of voice" and are told that their "facial expression is stiff," they can use that improvement in their next training session.

[0058] Step 8:

[0059] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve themselves and their skills. Users can repeat this process to hone their sales techniques.

[0060] The above are the detailed processing steps in the system of the present invention, which allows users to efficiently learn sales techniques and improve themselves.

[0061] Example 1

[0062] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0063] There is a need for a more efficient and standardized way to learn sales techniques and improve skills. However, current systems lack effective feedback and provide users with no easy way to evaluate and improve their performance. Furthermore, they lack advanced technology for analyzing video and audio data, resulting in limited quality of feedback.

[0064] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0065] In this invention, the server includes means for building a database for storing data on sales techniques, means for videotaping and recording a user's sales performance, means for transmitting the videotaped and recorded data to the server, means for the server to analyze the videotaped and recorded data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for the server to use a deep learning model for the analysis, means for the user terminal to upload collected performance data to the server, and means for the user to receive feedback in real time via the terminal. This allows users to utilize advanced technology to constantly analyze their sales performance and receive instant feedback, enabling efficient self-learning and skill improvement.

[0066] "Data on sales techniques" refers to information stored in a database that includes information on sales activity scenarios, techniques, methods, facial expressions, tone and volume of voice, etc.

[0067] "User's sales performance" refers to the records of the user's actual sales presentations and interactions with customers.

[0068] "Video and audio data" refers to video and audio data recorded using a camera or microphone of a user's sales performance.

[0069] "Server" is a central processing unit that stores, analyzes, and generates feedback on sales methodology data.

[0070] A "deep learning model" is a type of artificial intelligence, specifically a technology that uses deep neural networks to analyze data and recognize patterns.

[0071] "Feedback" is information that provides an evaluation of the user's sales performance and areas for improvement based on the results of analysis by the server.

[0072] "User terminal" means a device that allows a user to record and record sales performance and receive feedback.

[0073] A "prompt sentence" is an instruction sentence that prompts the user to take a specific action or input.

[0074] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is implemented by the following procedure and configuration.

[0075] System configuration and hardware / software

[0076] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0077] Users can use the device to record and record their sales performance. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later.

[0078] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0079] The server analyzes the received performance data. Here, advanced analysis technologies such as deep learning models (specifically, Tensorflow® and PyTorch) are used to evaluate the user's sales technique, facial expression, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify expressions such as a "serious expression" or a "smiling smile." It also analyzes voice data to identify expressions such as a "calm tone" or an "enthusiastic tone."

[0080] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0081] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0082] Specific examples

[0083] For example, consider a case where a user performs during a business negotiation using a "serious expression" and "calm tone." In this case, the server's deep learning model analyzes the performance and provides specific feedback such as "You should smile more." If the tone of voice is appropriate, the system will give a high rating of "Voice tone is very good." This allows the user to clearly understand their strengths and areas for improvement and use this knowledge in their next performance.

[0084] Prompt Sentence Examples

[0085] Examples of prompts include:

[0086] "Please provide the analysis results of facial expressions and tone of voice at the progress stage of a sales negotiation, along with specific advice. Please give us feedback when users use a serious facial expression and a calm tone of voice."

[0087] In this way, the system of the present invention supports efficient learning of sales techniques and the leveling of skills, and is a powerful tool for users to improve themselves.

[0088] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0089] Step 1:

[0090] The server builds a database of sales techniques. The input includes success stories and model scenarios related to sales activities. The server analyzes this information and saves detailed information such as sales scenarios, facial expressions, and tone of voice in the database. Specifically, the sales scenarios include "initial greetings with customers" and "progress of sales negotiations." The output is model sales scenarios saved in the database.

[0091] Step 2:

[0092] Users use a device to record and record their sales performance. Input includes presentations and conversations with customers that the user makes using the device's camera and microphone. The device records and saves these performances as data. As a specific example, consider the case where a user simulates a sales negotiation. The output is the video and audio data that is saved on the device.

[0093] Step 3:

[0094] After the recording is complete, the device sends the collected performance data to the server. The input includes the recording data. The device displays a "Send" button, and when the user presses this button, the data is uploaded to the server. The output is the recording data stored on the server and prepared for analysis.

[0095] Step 4:

[0096] The server analyzes the received performance data using a deep learning model. Input includes video and audio data. The server uses the deep learning model to evaluate the user's facial expressions, tone of voice, etc. Specifically, it analyzes facial movements to identify "serious expressions" and "smiling faces." It also analyzes voice data to identify "calm tones" and "passionate tones." The output generates evaluation data of the user's sales techniques, facial expressions, and tone of voice.

[0097] Step 5:

[0098] The server generates feedback based on the analysis results. The input includes the analysis results. In addition to evaluating the sales technique, the server generates feedback that includes suggestions for improvement in facial expressions and tone of voice. Specific feedback might include advice such as, "Your facial expression seems a little stiff, so next time try to smile more relaxedly." The output is feedback data.

[0099] Step 6:

[0100] The generated feedback is sent from the server to the user terminal. The input includes feedback data. The server sends the feedback to the user terminal, and the user can receive the feedback in real time. The output is displayed on the user terminal.

[0101] Step 7:

[0102] The user checks the feedback on the device and uses it as a reference for improving their sales skills. The input includes the feedback. The user reviews the feedback and creates a specific action plan to help their next sales performance. Specifically, a goal might be set, such as "In the next presentation, I will speak in a brighter tone and smile more." The output is a specific action plan to improve the user's skills.

[0103] (Application example 1)

[0104] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional sales technique learning systems often required users to rely on self-evaluation, making it difficult to receive objective feedback. Furthermore, face-to-face instruction was required, which often meant location and time constraints. Furthermore, it was difficult to evaluate small details such as facial expressions and tone of voice, making it difficult to standardize skills. This resulted in issues such as the time it took for new employees to become immediately productive, delaying the improvement of the skills of the entire sales department.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0107] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for the server to generate feedback based on the analysis results, means for transmitting the feedback to a user terminal and for the user to confirm, and means for providing training scenarios using prompt sentences. This allows users to receive objective and detailed feedback and improve their sales skills efficiently without being restricted by location or time.

[0108] The "database for storing data on sales techniques" is an information infrastructure for systematically storing information related to sales performance, such as model sales scenarios, effective facial expressions, and appropriate tone and volume of voice.

[0109] "Means for recording video and audio of a user's sales performance" refers to a device or system for recording a user's sales activities using devices such as a camera and microphone.

[0110] "Means for transmitting video and audio data to a server" refers to a communication device or program for uploading video and audio data to a server via a network.

[0111] "Means for the server to analyze video and audio data and evaluate sales techniques, facial expressions, and tone of voice" refers to technology that allows the server to analyze video and audio data using deep learning models, etc., and evaluate sales techniques, the user's facial expressions, and tone of voice.

[0112] The "means for the server to generate feedback based on the analysis results" refers to a system or program for automatically generating feedback indicating areas for improvement and strengths to the user based on the analysis results.

[0113] The "means for transmitting feedback to the user terminal and for the user to confirm" refers to the communication technology and user interface for delivering the feedback generated by the server to the user's terminal and allowing the user to view it.

[0114] The "means for providing a training scenario using a prompt sentence" is a system that prepares sentences in advance to present a scenario for the user's training and displays them to the user.

[0115] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is actually operated with the cooperation of a server, terminals, and users.

[0116] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0117] Next, users can use a device to record and record their sales performance. Using a camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. Devices used for this purpose include smartphones, tablets, and smart glasses.

[0118] After the recording is complete, the device sends the collected performance data to the server. The recorded data is then uploaded to the server via the network. The communication protocol and data format are assumed to be the standard HTTP / S and JSON.

[0119] The server analyzes the received performance data. This analysis uses advanced analytical techniques such as deep learning models. For example, it builds models using libraries such as TensorFlow and PyTorch to evaluate the user's sales technique, facial expressions, and tone of voice. Specifically, it analyzes the user's facial movements to identify expressions such as a "serious expression" or a "smiling face," and analyzes voice data to identify expressions such as a "calm tone" or an "passionate tone."

[0120] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0121] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0122] For example, when a user starts a workout, a prompt is displayed, such as:

[0123] "Please handle the following scenarios with your customers:

[0124] 1. Handling customer complaints

[0125] 2. The customer is angry

[0126] 3. Maintain a calm tone of voice and a polite facial expression.

[0127] After displaying this prompt, the user records and films their response, and the data is sent to a server for analysis and feedback.

[0128] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0129] Step 1:

[0130] The server builds a database of sales techniques.

[0131] Input: Data such as model sales scenarios, effective facial expressions, appropriate voice tone and volume, etc.

[0132] Output: Sales technique information stored in the database.

[0133] How it works: The server collects new sales data and stores it in a database, where it is categorized, tagged, and stored in an easy-to-search format.

[0134] Step 2:

[0135] Users can use the device to record and record their sales performance.

[0136] Input: Camera and microphone to record sales performance.

[0137] Output: Recorded and audio data saved on the device.

[0138] How it works: The user uses the device's camera and microphone to record and record their sales activities. Once the recording is complete, the data is temporarily saved on the device.

[0139] Step 3:

[0140] The terminal transmits the collected performance data to the server.

[0141] Input: Video and audio data stored on the device.

[0142] Output: Performance data uploaded to the server.

[0143] Operation: The device sends video and audio data to the server via the network. The data is sent using the HTTP / S protocol. The server receives the data and stores it for analysis.

[0144] Step 4:

[0145] The server analyzes the video and audio data and evaluates sales techniques, facial expressions, and tone of voice.

[0146] Input: Video and audio data stored on the server.

[0147] Output: Evaluation results on sales techniques, facial expressions, and tone of voice.

[0148] How it works: The server analyzes the data using deep learning models (such as TensorFlow or PyTorch), using image analysis to analyze facial expressions and audio analysis to evaluate tone of voice.

[0149] Step 5:

[0150] The server generates feedback based on the analysis results.

[0151] Input: Evaluation results for sales technique, facial expression, and tone of voice.

[0152] Output: The generated feedback information.

[0153] How it works: Based on the analysis results, the server generates feedback that indicates the user's sales skills and areas for improvement. The feedback is designed to include specific advice and evaluation.

[0154] Step 6:

[0155] The server transmits the generated feedback to the user terminal.

[0156] Input: The generated feedback information.

[0157] Output: Feedback displayed on the user's terminal.

[0158] Operation: The server sends the generated feedback information to the user's terminal via the network, where the user can view the feedback on the terminal.

[0159] Step 7:

[0160] Provide training scenarios using prompts.

[0161] Input: prompt, training scenario.

[0162] Output: The training scenario presented to the user.

[0163] Operation: The server selects a training scenario from the database and sends it to the user's device as a prompt. The user then follows the prompt to train. "Please handle the following customer situations: 1. Handling a customer complaint. 2. The customer is angry. 3. Maintain a calm tone of voice and a polite facial expression."

[0164] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0165] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[0166] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0167] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0168] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0169] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[0170] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0171] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0172] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0173] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[0174] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[0175] The processing flow will be explained below.

[0176] Step 1:

[0177] The server creates a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tones and volumes. For example, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0178] Step 2:

[0179] Users can record their sales performance using the device. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0180] Step 3:

[0181] The device transmits the recorded performance data to a server, where it is stored and prepared for analysis using a data upload function via the internet.

[0182] Step 4:

[0183] The server analyzes the received data, using highly accurate analytical methods such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling smile." It also analyzes audio data to identify whether a person has a "calm tone" or an "passionate tone."

[0184] Step 5:

[0185] The server then uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it analyzes changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0186] Step 6:

[0187] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0188] Step 7:

[0189] The server sends the generated feedback to the terminal, which provides the feedback in a format that is easy for the user to view and can check it on the terminal.

[0190] Step 8:

[0191] Users can check the feedback on their devices and use it as a reference for improving their sales skills. This allows users to understand areas for improvement based on the feedback and continue to improve themselves.

[0192] Step 9:

[0193] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve their skills. By repeating this process, users can hone their sales techniques and put them to practical use.

[0194] Example 2

[0195] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] There is a need for a system that can efficiently learn and standardize sales techniques and provide advanced feedback that takes users' emotions into account. However, existing systems have limited ability to evaluate users' sales performance, and in particular lack feedback that takes users' emotions into account. This makes it difficult for users to recognize fluctuations in their performance due to their own emotions and use them to improve.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0198] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for using an emotion engine to analyze the user's emotions, and means for reflecting the emotion analysis results in the generated feedback. This allows the user to receive feedback that takes into account both their sales performance and emotions, enabling more effective learning and skill improvement.

[0199] "Sales technique data" refers to data that includes detailed information such as specific scenarios related to sales processes and customer interactions, facial expressions, tone of voice, and volume.

[0200] A "database" is a collection of data related to sales techniques that is structured and stored so that it can be easily searched and referenced.

[0201] "User" means an individual or organization that intends to use this system to improve their own sales performance.

[0202] "Means for recording and recording sales performance" includes cameras and microphones for recording the user's sales activities as video and audio, as well as devices and programs for controlling their operation.

[0203] "Video and audio data" refers to video and audio data that records the user's business activities.

[0204] A "server" is a computer system that performs central processing such as storing data, analyzing data, and generating feedback.

[0205] A "deep learning model" is a type of artificial intelligence that uses neural networks to learn patterns from large amounts of data and perform analysis and classification.

[0206] The "emotion engine" is software that analyzes and recognizes a user's emotional state based on their video and audio data.

[0207] "Feedback" refers to evaluations and suggestions for improvement provided based on the analysis results of the sales performance recorded by the user.

[0208] "User terminal" means a device used by a user to record video and audio, and to receive feedback from the server.

[0209] "Analysis results" refer to information on evaluations and emotional states obtained from analysis by the video and audio data and the emotion engine.

[0210] "Means for evaluating sales techniques, facial expressions, and tone of voice" refers to the process in which the server analyzes video and audio data using deep learning models, etc., and evaluates the sales techniques, facial expressions, and tone of voice as a result.

[0211] "Means for reflecting the emotion analysis results in the generated feedback" refers to the process of creating detailed feedback that includes the user's emotional state based on the analysis results from the emotion engine.

[0212] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[0213] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0214] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0215] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0216] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[0217] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0218] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0219] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0220] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[0221] Example prompt: "We've recorded your performance in reaching new customers. We'd like you to analyze this data and provide feedback on how we can improve."

[0222] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[0223] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0224] Step 1:

[0225] The server builds a sales technique database.

[0226] Input: Information related to sales techniques such as sales scenarios, facial expression data, and tone of voice data.

[0227] Data processing: The information related to these sales techniques is stored in a database in a structured format.

[0228] Output: A database of sales techniques.

[0229] Specific operation: The server collects sales scenarios, facial expression data, and voice tone data, and stores them in a database. For example, it stores a scenario called "approaching a new customer."

[0230] Step 2:

[0231] Users can record and record their sales performance.

[0232] Input: User's actual sales performance (video and audio).

[0233] Data processing: The user's sales performance is recorded using a camera and microphone.

[0234] Output: Video and audio recording of performance data.

[0235] Specific operation: The user presses the record button on the device and performs a mock sales scenario. The prompt sentence is "Please record the mock sales scenario."

[0236] Step 3:

[0237] The terminal transmits the collected performance data to the server.

[0238] Input: Video and audio recording of performance data.

[0239] Data processing: After recording and audio recording, the data file is uploaded to the server via the network.

[0240] Output: Performance data sent to the server.

[0241] Specific operation: After recording and recording is completed, the data is transferred to the server and a message is displayed saying "Recording is complete. Data will be sent to the server."

[0242] Step 4:

[0243] The server analyzes the data.

[0244] Input: Transmitted performance data (video and audio).

[0245] Data processing: Deep learning models are used to analyze performance data and evaluate sales techniques, facial expressions, and tone of voice.

[0246] Output: Evaluation data on sales techniques, facial expressions, and tone of voice.

[0247] Specific operations: Analyzes facial movements from video data to identify "smiling faces," etc. Analyzes audio data to identify "calm tones."

[0248] Step 5:

[0249] The server analyzes emotions using an emotion engine.

[0250] Input: Performance data (video and audio).

[0251] Data processing: Analyze user emotions from both video and audio, and reflect the results in the evaluation of sales performance.

[0252] Output: User sentiment analysis results.

[0253] Specific operation: Analyzes changes in the user's facial expression and voice to determine emotions such as tension or relaxation. Displays the message "Analyzing user's facial expression and tone of voice."

[0254] Step 6:

[0255] The server generates the feedback.

[0256] Input: Evaluation data of sales techniques, facial expressions, and tone of voice, as well as emotion analysis results.

[0257] Data processing: Generate feedback based on the evaluation data and also reflect the results of user sentiment analysis.

[0258] Output: The generated feedback.

[0259] Specific actions: Generate specific advice such as, "Your tone of voice is good, but you should try to look a little more relaxed with your facial expression."

[0260] Step 7:

[0261] The server transmits the generated feedback to the user terminal.

[0262] Input: The generated feedback.

[0263] Data processing: Send the feedback to the user's device and notify the user.

[0264] Output: Feedback displayed on the user's device.

[0265] What happens: Sends feedback to the device and notifies the user, "You have new feedback. Check it out."

[0266] (Application example 2)

[0267] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] In the case of conventional sales-based efficient learning systems, it is difficult to properly recognize users' emotions and reflect them in feedback, making it difficult for users to effectively improve their self-study and customer service skills.In addition, the lack of specific learning support for stores and customer service work makes it difficult for new employees in particular to immediately contribute.

[0269] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for building a database for storing data related to sales techniques, means for recording the user's sales performance, and means for transmitting the recorded data to the server. This makes it possible to comprehensively evaluate the user's sales techniques, facial expressions, tone of voice, and emotions, effectively supporting the learning of customer service skills in stores and customer service work. Furthermore, by providing specific feedback using the generative AI model, the user can clearly understand their strengths and areas for improvement, promoting continuous skill improvement.

[0270] "Sales techniques" are a series of processes and techniques from the first contact with a customer to the conclusion of a contract, and include effective facial expressions, tone of voice, scenarios, etc.

[0271] "Performance" refers to the specific actions and behaviors that users actually perform in sales activities and customer interactions.

[0272] "Video recording" refers to the use of a camera or microphone to record a user's voice, facial expressions, movements, etc. as video and audio data.

[0273] A "server" is a computer system for analyzing stored data and generating feedback.

[0274] "Analysis" is the process of analyzing video and audio data to evaluate the user's sales techniques and emotions.

[0275] "Feedback" refers to evaluations and suggestions or advice on areas for improvement provided to users based on the analysis results.

[0276] "User terminal" refers to a device such as a smartphone or computer used by a user to check feedback.

[0277] "Emotion recognition" is a technology that identifies a user's emotions from video and audio data, and determines their state of tension or relaxation.

[0278] A "generative AI model" is a computer model that uses artificial intelligence to automatically generate text and data.

[0279] "Store and customer service operations" refers to all service activities, including sales activities and customer support, in physical stores.

[0280] The system of this invention combines efficient learning of sales techniques with user emotion recognition to help improve customer service skills in stores and customer service work. This system consists of the following components and processing steps.

[0281] Hardware and software used

[0282] Device: Devices such as smartphones and head-mounted displays are used to record and record users' sales performance.

[0283] Server: A computer system responsible for storing data, analyzing data, and generating feedback, utilizing cloud services such as AWS® or Google® Cloud.

[0284] Deep learning model: Libraries such as TensorFlow and PyTorch are used for data analysis.

[0285] Emotion engine: Uses OpenFace, Microsoft® Azure® Emotion API, etc. to recognize emotions from video and audio data.

[0286] Generative AI model: Uses generative AI techniques to generate feedback statements.

[0287] System Program Processing

[0288] 1. Data Collection:

[0289] The device uses a camera and microphone to record and record the user's sales performance.

[0290] Example: Salespeople use their smartphones to record themselves practicing customer interactions.

[0291] 2. Data transmission:

[0292] The terminal transmits the collected data to the server.

[0293] Example: Recorded data is uploaded to a cloud server via Wi-Fi.

[0294] 3. Data Analysis:

[0295] A deep learning model is used on the server to analyze the recorded and audio data.

[0296] An emotion engine is used to identify the user's emotion.

[0297] Example: The server analyzes the user's facial expressions and tone of voice from video data and identifies "calm tone" and "tense expression."

[0298] 4. Feedback Generation:

[0299] Based on the analysis results, a generative AI model is used to generate specific feedback.

[0300] Example prompt sentence:

[0301] Feed your voice data into a deep learning model to identify patterns such as "calm tone," "passionate tone," and "friendly tone."

[0302] Analyze facial video data and identify expressions such as "smiling," "serious," "relaxed," and "tense."

[0303] Identify emotions from video and audio data and identify the emotions of salespeople when interacting with customers.

[0304] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[0305] Specific feedback examples:

[0306] Your tone of voice is very calm and puts the customer at ease. However, your facial expression is a little stiff, so try to relax and smile a little more. Your overall performance is good, and you have a very good foundation for building trust with the customer.

[0307] 5. Send Feedback:

[0308] The generated feedback is sent from the server to the user terminal, where the user can check it on the terminal.

[0309] Add specific examples to the description

[0310] For example, suppose a salesperson uses their smartphone to record a customer interaction practice. The recorded data is then sent to a server and analyzed using a deep learning model. Based on the analysis results, the generative AI model generates feedback such as "Your tone of voice is very good, but your facial expression is stiff," which is then sent to the user's device. The user can then review the feedback and use it in their next customer interaction.

[0311] This system allows users to clearly understand their strengths and areas for improvement, enabling continuous skill improvement. It also allows new employees, in particular, to quickly acquire customer service skills and become immediately effective.

[0312] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0313] Step 1: Data collection

[0314] Using a device (smartphone or head-mounted display), the user can record and record their sales performance. Specifically, a camera and microphone are used to record customer service practice and actual conversations. The input data is video and audio data, and the output is a video and audio file.

[0315] Step 2: Send data

[0316] The device sends the video and audio data collected in step 1 to the server. Specifically, it uploads the data using an HTTP request or WebSocket protocol over Wi-Fi or a mobile data network. The input is the video and audio file, and the output is the data stored on the server.

[0317] Step 3: Data analysis

[0318] The server analyzes the video and audio data it receives. In this step, a deep learning model is used to analyze facial expressions from the video data and tone of voice from the audio data. The input is the video and audio data, and the output is the analysis results: facial expression data, tone of voice data, and emotion recognition results. Specifically, a model using TensorFlow and PyTorch is run to extract features such as "calm tone," "serious expression," and "tension."

[0319] Step 4: Emotion Recognition

[0320] The server recognizes the user's emotions using an emotion engine (e.g., OpenFace or Microsoft Azure Emotion API) based on the analysis results from step 3. The input is facial expression data and tone of voice data, and the output is the user's emotional assessment (e.g., "relaxed" or "tense").

[0321] Step 5: Feedback generation

[0322] The server uses a generative AI model to generate specific feedback sentences based on the analysis results and emotion recognition results. Using example prompt sentences, feedback including evaluation points and areas for improvement is generated. The input is the analysis results and emotion evaluation data, and the output is the feedback sentence. Specifically, the following prompt sentences are used:

[0323] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[0324] Step 6: Send your feedback

[0325] The server sends the generated feedback sentence to the user device. In this step, a cloud messaging service (e.g., Firebase Cloud Messaging) is used to display the feedback sentence on the user device. The input is the feedback sentence, and the output is the feedback displayed on the user device.

[0326] Step 7: Check feedback and try again

[0327] The user checks the feedback on the device and attempts to perform the sales performance again based on the areas for improvement. Specifically, the user understands the feedback and records new conversations and customer responses. The input is the feedback text, and the output is video and audio data of the improved sales performance.

[0328] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0329] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0330] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0331] [Second embodiment]

[0332] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0333] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0334] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0335] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0336] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0337] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0338] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0339] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0340] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0341] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0342] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0343] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0344] The present invention is a system for efficiently learning sales techniques and leveling skills, and is implemented by the following procedure and configuration.

[0345] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0346] Next, users can use the device to record and record their sales performance. Users can use the camera and microphone to record presentations and conversations with customers, allowing them to review their sales techniques later.

[0347] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0348] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[0349] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0350] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0351] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, the user will receive feedback such as "You should smile more." If the tone of voice is appropriate, the user will be evaluated as "Very good tone of voice." This allows the user to clearly understand their strengths and areas for improvement and use this information in their next performance.

[0352] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and provides a powerful tool for users to improve themselves.

[0353] The processing flow will be explained below.

[0354] Step 1:

[0355] The server creates a database of sales techniques. Detailed information such as model sales scenarios, facial expressions, tone and volume of voice is entered into this database. For example, scenarios such as "initial greeting with a customer," "progression of a negotiation," and "successful closing" are saved.

[0356] Step 2:

[0357] Users can use their devices to record and record their sales performance. Using a camera and microphone, users can record their speech and facial expressions for later analysis. For example, they can conduct a mock sales meeting in front of the camera.

[0358] Step 3:

[0359] The device sends the recorded performance data to a server, where it is securely uploaded and stored. This is done using the data upload function via the internet.

[0360] Step 4:

[0361] The server analyzes the received data, using highly accurate analytical techniques such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling face."

[0362] Step 5:

[0363] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique and specific points for improvement such as facial expressions and tone of voice. For example, the server may provide advice such as, "Your tone of voice is good, but you should smile a bit more."

[0364] Step 6:

[0365] The server sends the generated feedback to the terminal, where it is displayed for the user to view. The feedback is provided in a format that is easy for the user to view.

[0366] Step 7:

[0367] Users check the feedback on their device. Based on the feedback provided, they can understand the strengths and areas for improvement in their sales performance. For example, if they are evaluated as having an "appropriate tone of voice" and are told that their "facial expression is stiff," they can use that improvement in their next training session.

[0368] Step 8:

[0369] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve themselves and their skills. Users can repeat this process to hone their sales techniques.

[0370] The above are the detailed processing steps in the system of the present invention, which allows users to efficiently learn sales techniques and improve themselves.

[0371] Example 1

[0372] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0373] There is a need for a more efficient and standardized way to learn sales techniques and improve skills. However, current systems lack effective feedback and provide users with no easy way to evaluate and improve their performance. Furthermore, they lack advanced technology for analyzing video and audio data, resulting in limited quality of feedback.

[0374] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0375] In this invention, the server includes means for building a database for storing data on sales techniques, means for videotaping and recording a user's sales performance, means for transmitting the videotaped and recorded data to the server, means for the server to analyze the videotaped and recorded data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for the server to use a deep learning model for the analysis, means for the user terminal to upload collected performance data to the server, and means for the user to receive feedback in real time via the terminal. This allows users to utilize advanced technology to constantly analyze their sales performance and receive instant feedback, enabling efficient self-learning and skill improvement.

[0376] "Data on sales techniques" refers to information stored in a database that includes information on sales activity scenarios, techniques, methods, facial expressions, tone and volume of voice, etc.

[0377] "User's sales performance" refers to the records of the user's actual sales presentations and interactions with customers.

[0378] "Video and audio data" refers to video and audio data recorded using a camera or microphone of a user's sales performance.

[0379] "Server" is a central processing unit that stores, analyzes, and generates feedback on sales methodology data.

[0380] A "deep learning model" is a type of artificial intelligence, specifically a technology that uses deep neural networks to analyze data and recognize patterns.

[0381] "Feedback" is information that provides an evaluation of the user's sales performance and areas for improvement based on the results of analysis by the server.

[0382] "User terminal" means a device that allows a user to record and record sales performance and receive feedback.

[0383] A "prompt sentence" is an instruction sentence that prompts the user to take a specific action or input.

[0384] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is implemented by the following procedure and configuration.

[0385] System configuration and hardware / software

[0386] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0387] Users can use the device to record and record their sales performance. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later.

[0388] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0389] The server analyzes the received performance data. Here, advanced analysis techniques such as deep learning models (specifically TensorFlow and PyTorch) are used to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[0390] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0391] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0392] Specific examples

[0393] For example, consider a case where a user performs during a business negotiation using a "serious expression" and "calm tone." In this case, the server's deep learning model analyzes the performance and provides specific feedback such as "You should smile more." If the tone of voice is appropriate, the system will give a high rating of "Voice tone is very good." This allows the user to clearly understand their strengths and areas for improvement and use this knowledge in their next performance.

[0394] Prompt Sentence Examples

[0395] Examples of prompts include:

[0396] "Please provide the analysis results of facial expressions and tone of voice at the progress stage of a sales negotiation, along with specific advice. Please give us feedback when users use a serious facial expression and a calm tone of voice."

[0397] In this way, the system of the present invention supports efficient learning of sales techniques and the leveling of skills, and is a powerful tool for users to improve themselves.

[0398] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0399] Step 1:

[0400] The server builds a database of sales techniques. The input includes success stories and model scenarios related to sales activities. The server analyzes this information and saves detailed information such as sales scenarios, facial expressions, and tone of voice in the database. Specifically, the sales scenarios include "initial greetings with customers" and "progress of sales negotiations." The output is model sales scenarios saved in the database.

[0401] Step 2:

[0402] Users use a device to record and record their sales performance. Input includes presentations and conversations with customers that the user makes using the device's camera and microphone. The device records and saves these performances as data. As a specific example, consider the case where a user simulates a sales negotiation. The output is the video and audio data that is saved on the device.

[0403] Step 3:

[0404] After the recording is complete, the device sends the collected performance data to the server. The input includes the recording data. The device displays a "Send" button, and when the user presses this button, the data is uploaded to the server. The output is the recording data stored on the server and prepared for analysis.

[0405] Step 4:

[0406] The server analyzes the received performance data using a deep learning model. Input includes video and audio data. The server uses the deep learning model to evaluate the user's facial expressions, tone of voice, etc. Specifically, it analyzes facial movements to identify "serious expressions" and "smiling faces." It also analyzes voice data to identify "calm tones" and "passionate tones." The output generates evaluation data of the user's sales techniques, facial expressions, and tone of voice.

[0407] Step 5:

[0408] The server generates feedback based on the analysis results. The input includes the analysis results. In addition to evaluating the sales technique, the server generates feedback that includes suggestions for improvement in facial expressions and tone of voice. Specific feedback might include advice such as, "Your facial expression seems a little stiff, so next time try to smile more relaxedly." The output is feedback data.

[0409] Step 6:

[0410] The generated feedback is sent from the server to the user terminal. The input includes feedback data. The server sends the feedback to the user terminal, and the user can receive the feedback in real time. The output is displayed on the user terminal.

[0411] Step 7:

[0412] The user checks the feedback on the device and uses it as a reference for improving their sales skills. The input includes the feedback. The user reviews the feedback and creates a specific action plan to help their next sales performance. Specifically, a goal might be set, such as "In the next presentation, I will speak in a brighter tone and smile more." The output is a specific action plan to improve the user's skills.

[0413] (Application example 1)

[0414] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0415] Conventional sales technique learning systems often required users to rely on self-evaluation, making it difficult to receive objective feedback. Furthermore, face-to-face instruction was required, which often meant location and time constraints. Furthermore, it was difficult to evaluate small details such as facial expressions and tone of voice, making it difficult to standardize skills. This resulted in issues such as the time it took for new employees to become immediately productive, delaying the improvement of the skills of the entire sales department.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0417] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for the server to generate feedback based on the analysis results, means for transmitting the feedback to a user terminal and for the user to confirm, and means for providing training scenarios using prompt sentences. This allows users to receive objective and detailed feedback and improve their sales skills efficiently without being restricted by location or time.

[0418] The "database for storing data on sales techniques" is an information infrastructure for systematically storing information related to sales performance, such as model sales scenarios, effective facial expressions, and appropriate tone and volume of voice.

[0419] "Means for recording video and audio of a user's sales performance" refers to a device or system for recording a user's sales activities using devices such as a camera and microphone.

[0420] "Means for transmitting video and audio data to a server" refers to a communication device or program for uploading video and audio data to a server via a network.

[0421] "Means for the server to analyze video and audio data and evaluate sales techniques, facial expressions, and tone of voice" refers to technology that allows the server to analyze video and audio data using deep learning models, etc., and evaluate sales techniques, the user's facial expressions, and tone of voice.

[0422] The "means for the server to generate feedback based on the analysis results" refers to a system or program for automatically generating feedback indicating areas for improvement and strengths to the user based on the analysis results.

[0423] The "means for transmitting feedback to the user terminal and for the user to confirm" refers to the communication technology and user interface for delivering the feedback generated by the server to the user's terminal and allowing the user to view it.

[0424] The "means for providing a training scenario using a prompt sentence" is a system that prepares sentences in advance to present a scenario for the user's training and displays them to the user.

[0425] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is actually operated with the cooperation of a server, terminals, and users.

[0426] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0427] Next, users can use a device to record and record their sales performance. Using a camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. Devices used for this purpose include smartphones, tablets, and smart glasses.

[0428] After the recording is complete, the device sends the collected performance data to the server. The recorded data is then uploaded to the server via the network. The communication protocol and data format are assumed to be the standard HTTP / S and JSON.

[0429] The server analyzes the received performance data. This analysis uses advanced analytical techniques such as deep learning models. For example, it builds models using libraries such as TensorFlow and PyTorch to evaluate the user's sales technique, facial expressions, and tone of voice. Specifically, it analyzes the user's facial movements to identify expressions such as a "serious expression" or a "smiling face," and analyzes voice data to identify expressions such as a "calm tone" or an "passionate tone."

[0430] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0431] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0432] For example, when a user starts a workout, a prompt is displayed, such as:

[0433] "Please handle the following scenarios with your customers:

[0434] 1. Handling customer complaints

[0435] 2. The customer is angry

[0436] 3. Maintain a calm tone of voice and a polite facial expression.

[0437] After displaying this prompt, the user records and films their response, and the data is sent to a server for analysis and feedback.

[0438] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0439] Step 1:

[0440] The server builds a database of sales techniques.

[0441] Input: Data such as model sales scenarios, effective facial expressions, appropriate voice tone and volume, etc.

[0442] Output: Sales technique information stored in the database.

[0443] How it works: The server collects new sales data and stores it in a database, where it is categorized, tagged, and stored in an easy-to-search format.

[0444] Step 2:

[0445] Users can use the device to record and record their sales performance.

[0446] Input: Camera and microphone to record sales performance.

[0447] Output: Recorded and audio data saved on the device.

[0448] How it works: The user uses the device's camera and microphone to record and record their sales activities. Once the recording is complete, the data is temporarily saved on the device.

[0449] Step 3:

[0450] The terminal transmits the collected performance data to the server.

[0451] Input: Video and audio data stored on the device.

[0452] Output: Performance data uploaded to the server.

[0453] Operation: The device sends video and audio data to the server via the network. The data is sent using the HTTP / S protocol. The server receives the data and stores it for analysis.

[0454] Step 4:

[0455] The server analyzes the video and audio data and evaluates sales techniques, facial expressions, and tone of voice.

[0456] Input: Video and audio data stored on the server.

[0457] Output: Evaluation results on sales techniques, facial expressions, and tone of voice.

[0458] How it works: The server analyzes the data using deep learning models (such as TensorFlow or PyTorch), using image analysis to analyze facial expressions and audio analysis to evaluate tone of voice.

[0459] Step 5:

[0460] The server generates feedback based on the analysis results.

[0461] Input: Evaluation results for sales technique, facial expression, and tone of voice.

[0462] Output: The generated feedback information.

[0463] How it works: Based on the analysis results, the server generates feedback that indicates the user's sales skills and areas for improvement. The feedback is designed to include specific advice and evaluation.

[0464] Step 6:

[0465] The server transmits the generated feedback to the user terminal.

[0466] Input: The generated feedback information.

[0467] Output: Feedback displayed on the user's terminal.

[0468] Operation: The server sends the generated feedback information to the user's terminal via the network, where the user can view the feedback on the terminal.

[0469] Step 7:

[0470] Provide training scenarios using prompts.

[0471] Input: prompt, training scenario.

[0472] Output: The training scenario presented to the user.

[0473] Operation: The server selects a training scenario from the database and sends it to the user's device as a prompt. The user then follows the prompt to train. "Please handle the following customer situations: 1. Handling a customer complaint. 2. The customer is angry. 3. Maintain a calm tone of voice and a polite facial expression."

[0474] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0475] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[0476] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0477] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0478] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0479] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[0480] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0481] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0482] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0483] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[0484] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[0485] The processing flow will be explained below.

[0486] Step 1:

[0487] The server creates a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tones and volumes. For example, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0488] Step 2:

[0489] Users can record their sales performance using the device. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0490] Step 3:

[0491] The device transmits the recorded performance data to a server, where it is stored and prepared for analysis using a data upload function via the internet.

[0492] Step 4:

[0493] The server analyzes the received data, using highly accurate analytical methods such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling smile." It also analyzes audio data to identify whether a person has a "calm tone" or an "passionate tone."

[0494] Step 5:

[0495] The server then uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it analyzes changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0496] Step 6:

[0497] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0498] Step 7:

[0499] The server sends the generated feedback to the terminal, which provides the feedback in a format that is easy for the user to view and can check it on the terminal.

[0500] Step 8:

[0501] Users can check the feedback on their devices and use it as a reference for improving their sales skills. This allows users to understand areas for improvement based on the feedback and continue to improve themselves.

[0502] Step 9:

[0503] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve their skills. By repeating this process, users can hone their sales techniques and put them to practical use.

[0504] Example 2

[0505] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0506] There is a need for a system that can efficiently learn and standardize sales techniques and provide advanced feedback that takes users' emotions into account. However, existing systems have limited ability to evaluate users' sales performance, and in particular lack feedback that takes users' emotions into account. This makes it difficult for users to recognize fluctuations in their performance due to their own emotions and use them to improve.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0508] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales technique, facial expression, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for using an emotion engine to analyze the user's emotions, and means for reflecting the emotion analysis results in the generated feedback. This allows the user to receive feedback that takes into account both their sales performance and emotions, enabling more effective learning and skill improvement.

[0509] "Sales technique data" refers to data that includes detailed information such as specific scenarios related to sales processes and customer interactions, facial expressions, tone of voice, and volume.

[0510] A "database" is a collection of data related to sales techniques that is structured and stored so that it can be easily searched and referenced.

[0511] "User" means an individual or organization that intends to use this system to improve their own sales performance.

[0512] "Means for recording and recording sales performance" includes cameras and microphones for recording the user's sales activities as video and audio, as well as devices and programs for controlling their operation.

[0513] "Video and audio data" refers to video and audio data that records the user's business activities.

[0514] A "server" is a computer system that performs central processing such as storing data, analyzing data, and generating feedback.

[0515] A "deep learning model" is a type of artificial intelligence that uses neural networks to learn patterns from large amounts of data and perform analysis and classification.

[0516] The "emotion engine" is software that analyzes and recognizes a user's emotional state based on their video and audio data.

[0517] "Feedback" refers to evaluations and suggestions for improvement provided based on the analysis results of the sales performance recorded by the user.

[0518] "User terminal" means a device used by a user to record video and audio, and to receive feedback from the server.

[0519] "Analysis results" refer to information on evaluations and emotional states obtained from analysis by the video and audio data and the emotion engine.

[0520] "Means for evaluating sales techniques, facial expressions, and tone of voice" refers to the process in which the server analyzes video and audio data using deep learning models, etc., and evaluates the sales techniques, facial expressions, and tone of voice as a result.

[0521] "Means for reflecting the emotion analysis results in the generated feedback" refers to the process of creating detailed feedback that includes the user's emotional state based on the analysis results from the emotion engine.

[0522] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[0523] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0524] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0525] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0526] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[0527] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0528] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0529] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0530] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[0531] Example prompt: "We've recorded your performance in reaching new customers. We'd like you to analyze this data and provide feedback on how we can improve."

[0532] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[0533] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0534] Step 1:

[0535] The server builds a sales technique database.

[0536] Input: Information related to sales techniques such as sales scenarios, facial expression data, and tone of voice data.

[0537] Data processing: The information related to these sales techniques is stored in a database in a structured format.

[0538] Output: A database of sales techniques.

[0539] Specific operation: The server collects sales scenarios, facial expression data, and voice tone data, and stores them in a database. For example, it stores a scenario called "Approaching a new customer."

[0540] Step 2:

[0541] Users can record and record their sales performance.

[0542] Input: User's actual sales performance (video and audio).

[0543] Data processing: The user's sales performance is recorded using a camera and microphone.

[0544] Output: Video and audio recording of performance data.

[0545] Specific operation: The user presses the record button on the device and performs a mock sales scenario. The prompt sentence is "Please record and record the mock sales scenario."

[0546] Step 3:

[0547] The terminal transmits the collected performance data to the server.

[0548] Input: Video and audio recording of performance data.

[0549] Data processing: After recording and audio recording, the data file is uploaded to the server via the network.

[0550] Output: Performance data sent to the server.

[0551] Specific operation: After recording and recording is completed, the data is transferred to the server and a message is displayed saying "Recording is complete. Data will be sent to the server."

[0552] Step 4:

[0553] The server analyzes the data.

[0554] Input: Transmitted performance data (video and audio).

[0555] Data processing: Deep learning models are used to analyze performance data and evaluate sales techniques, facial expressions, and tone of voice.

[0556] Output: Evaluation data on sales techniques, facial expressions, and tone of voice.

[0557] Specific operations: Analyzes facial movements from video data to identify "smiling faces," etc. Analyzes audio data to identify "calm tones."

[0558] Step 5:

[0559] The server analyzes emotions using an emotion engine.

[0560] Input: Performance data (video and audio).

[0561] Data processing: Analyze user emotions from both video and audio, and reflect the results in the evaluation of sales performance.

[0562] Output: User sentiment analysis results.

[0563] Specific operation: Analyzes changes in the user's facial expression and voice to determine emotions such as tension or relaxation. Displays the message "Analyzing user's facial expression and tone of voice."

[0564] Step 6:

[0565] The server generates the feedback.

[0566] Input: Evaluation data of sales techniques, facial expressions, and tone of voice, as well as emotion analysis results.

[0567] Data processing: Generate feedback based on the evaluation data and also reflect the results of user sentiment analysis.

[0568] Output: The generated feedback.

[0569] Specific actions: Generate specific advice such as, "Your tone of voice is good, but you should try to look a little more relaxed with your facial expression."

[0570] Step 7:

[0571] The server transmits the generated feedback to the user terminal.

[0572] Input: The generated feedback.

[0573] Data processing: Send the feedback to the user's device and notify the user.

[0574] Output: Feedback displayed on the user's device.

[0575] What happens: Sends feedback to the device and notifies the user, "You have new feedback. Check it out."

[0576] (Application example 2)

[0577] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] In the case of conventional sales-based efficient learning systems, it is difficult to properly recognize users' emotions and reflect them in feedback, making it difficult for users to effectively improve their self-study and customer service skills.In addition, the lack of specific learning support for stores and customer service work makes it difficult for new employees in particular to immediately contribute.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for building a database for storing data related to sales techniques, means for recording the user's sales performance, and means for transmitting the recorded data to the server. This makes it possible to comprehensively evaluate the user's sales techniques, facial expressions, tone of voice, and emotions, effectively supporting the learning of customer service skills in stores and customer service work. Furthermore, by providing specific feedback using the generative AI model, the user can clearly understand their strengths and areas for improvement, promoting continuous skill improvement.

[0580] "Sales techniques" are a series of processes and techniques from the first contact with a customer to the conclusion of a contract, and include effective facial expressions, tone of voice, scenarios, etc.

[0581] "Performance" refers to the specific actions and behaviors that users actually perform in sales activities and customer interactions.

[0582] "Video recording" refers to the use of a camera or microphone to record a user's voice, facial expressions, movements, etc. as video and audio data.

[0583] A "server" is a computer system for analyzing stored data and generating feedback.

[0584] "Analysis" is the process of analyzing video and audio data to evaluate the user's sales techniques and emotions.

[0585] "Feedback" refers to evaluations and suggestions or advice on areas for improvement provided to users based on the analysis results.

[0586] "User terminal" refers to a device such as a smartphone or computer used by a user to check feedback.

[0587] "Emotion recognition" is a technology that identifies a user's emotions from video and audio data, and determines their state of tension or relaxation.

[0588] A "generative AI model" is a computer model that uses artificial intelligence to automatically generate text and data.

[0589] "Store and customer service operations" refers to all service activities, including sales activities and customer support, in physical stores.

[0590] The system of this invention combines efficient learning of sales techniques with user emotion recognition to help improve customer service skills in stores and customer service work. This system consists of the following components and processing steps.

[0591] Hardware and software used

[0592] Device: Devices such as smartphones and head-mounted displays are used to record and record users' sales performance.

[0593] Server: A computer system responsible for storing data, analyzing it, and generating feedback. It utilizes cloud services such as AWS or Google Cloud.

[0594] Deep learning model: Libraries such as TensorFlow and PyTorch are used for data analysis.

[0595] Emotion engine: Uses OpenFace, Microsoft Azure Emotion API, etc. to recognize emotions from video and audio data.

[0596] Generative AI model: Uses generative AI techniques to generate feedback statements.

[0597] System Program Processing

[0598] 1. Data Collection:

[0599] The device uses a camera and microphone to record and record the user's sales performance.

[0600] Example: Salespeople use their smartphones to record themselves practicing customer interactions.

[0601] 2. Data transmission:

[0602] The terminal transmits the collected data to the server.

[0603] Example: Recorded data is uploaded to a cloud server via Wi-Fi.

[0604] 3. Data Analysis:

[0605] A deep learning model is used on the server to analyze the recorded and audio data.

[0606] An emotion engine is used to identify the user's emotion.

[0607] Example: The server analyzes the user's facial expressions and tone of voice from video data and identifies "calm tone" and "tense expression."

[0608] 4. Feedback Generation:

[0609] Based on the analysis results, a generative AI model is used to generate specific feedback.

[0610] Example prompt sentence:

[0611] Feed your voice data into a deep learning model to identify patterns such as "calm tone," "passionate tone," and "friendly tone."

[0612] Analyze facial video data and identify expressions such as "smiling," "serious," "relaxed," and "tense."

[0613] Identify emotions from video and audio data and identify the emotions of salespeople when interacting with customers.

[0614] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[0615] Specific feedback examples:

[0616] Your tone of voice is very calm and puts the customer at ease. However, your facial expression is a little stiff, so try to relax and smile a little more. Your overall performance is good, and you have a very good foundation for building trust with the customer.

[0617] 5. Send Feedback:

[0618] The generated feedback is sent from the server to the user terminal, where the user can check it on the terminal.

[0619] Add specific examples to the description

[0620] For example, suppose a salesperson uses their smartphone to record a customer interaction practice. The recorded data is then sent to a server and analyzed using a deep learning model. Based on the analysis results, the generative AI model generates feedback such as "Your tone of voice is very good, but your facial expression is stiff," which is then sent to the user's device. The user can then review the feedback and use it in their next customer interaction.

[0621] This system allows users to clearly understand their strengths and areas for improvement, enabling continuous skill improvement. It also allows new employees, in particular, to quickly acquire customer service skills and become immediately effective.

[0622] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0623] Step 1: Data collection

[0624] Using a device (smartphone or head-mounted display), the user can record and record their sales performance. Specifically, a camera and microphone are used to record customer service practice and actual conversations. The input data is video and audio data, and the output is a video and audio file.

[0625] Step 2: Send data

[0626] The device sends the video and audio data collected in step 1 to the server. Specifically, it uploads the data using an HTTP request or WebSocket protocol over Wi-Fi or a mobile data network. The input is the video and audio file, and the output is the data stored on the server.

[0627] Step 3: Data analysis

[0628] The server analyzes the video and audio data it receives. In this step, a deep learning model is used to analyze facial expressions from the video data and tone of voice from the audio data. The input is the video and audio data, and the output is the analysis results: facial expression data, tone of voice data, and emotion recognition results. Specifically, a model using TensorFlow and PyTorch is run to extract features such as "calm tone," "serious expression," and "tension."

[0629] Step 4: Emotion Recognition

[0630] The server recognizes the user's emotions using an emotion engine (e.g., OpenFace or Microsoft Azure Emotion API) based on the analysis results from step 3. The input is facial expression data and tone of voice data, and the output is the user's emotional assessment (e.g., "relaxed" or "tense").

[0631] Step 5: Feedback generation

[0632] The server uses a generative AI model to generate specific feedback sentences based on the analysis results and emotion recognition results. Using example prompt sentences, feedback including evaluation points and areas for improvement is generated. The input is the analysis results and emotion evaluation data, and the output is the feedback sentence. Specifically, the following prompt sentences are used:

[0633] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[0634] Step 6: Send your feedback

[0635] The server sends the generated feedback sentence to the user device. In this step, a cloud messaging service (e.g., Firebase Cloud Messaging) is used to display the feedback sentence on the user device. The input is the feedback sentence, and the output is the feedback displayed on the user device.

[0636] Step 7: Check feedback and try again

[0637] The user checks the feedback on the device and attempts to perform the sales performance again based on the areas for improvement. Specifically, the user understands the feedback and records new conversations and customer responses. The input is the feedback text, and the output is video and audio data of the improved sales performance.

[0638] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0639] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0640] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0641] [Third embodiment]

[0642] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0643] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0644] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0645] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0646] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0647] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0648] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0649] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0650] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0651] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0652] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0653] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0654] The present invention is a system for efficiently learning sales techniques and leveling skills, and is implemented by the following procedure and configuration.

[0655] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0656] Next, users can use the device to record and record their sales performance. Users can use the camera and microphone to record presentations and conversations with customers, allowing them to review their sales techniques later.

[0657] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0658] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[0659] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0660] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0661] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, the user will receive feedback such as "You should smile more." If the tone of voice is appropriate, the user will be evaluated as "Very good tone of voice." This allows the user to clearly understand their strengths and areas for improvement and use this information in their next performance.

[0662] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and provides a powerful tool for users to improve themselves.

[0663] The processing flow will be explained below.

[0664] Step 1:

[0665] The server creates a database of sales techniques. Detailed information such as model sales scenarios, facial expressions, tone and volume of voice is entered into this database. For example, scenarios such as "initial greeting with a customer," "progression of a negotiation," and "successful closing" are saved.

[0666] Step 2:

[0667] Users can use their devices to record and record their sales performance. Using a camera and microphone, users can record their speech and facial expressions for later analysis. For example, they can conduct a mock sales meeting in front of the camera.

[0668] Step 3:

[0669] The device sends the recorded performance data to a server, where it is securely uploaded and stored. This is done using the data upload function via the internet.

[0670] Step 4:

[0671] The server analyzes the received data, using highly accurate analytical techniques such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling face."

[0672] Step 5:

[0673] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique and specific points for improvement such as facial expressions and tone of voice. For example, the server may provide advice such as, "Your tone of voice is good, but you should smile a bit more."

[0674] Step 6:

[0675] The server sends the generated feedback to the terminal, where it is displayed for the user to view. The feedback is provided in a format that is easy for the user to view.

[0676] Step 7:

[0677] Users check the feedback on their device. Based on the feedback provided, they can understand the strengths and areas for improvement in their sales performance. For example, if they are evaluated as having an "appropriate tone of voice" and are told that their "facial expression is stiff," they can use that improvement in their next training session.

[0678] Step 8:

[0679] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve themselves and their skills. Users can repeat this process to hone their sales techniques.

[0680] The above are the detailed processing steps in the system of the present invention, which allows users to efficiently learn sales techniques and improve themselves.

[0681] Example 1

[0682] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0683] There is a need for a more efficient and standardized way to learn sales techniques and improve skills. However, current systems lack effective feedback and provide users with no easy way to evaluate and improve their performance. Furthermore, they lack advanced technology for analyzing video and audio data, resulting in limited quality of feedback.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0685] In this invention, the server includes means for building a database for storing data on sales techniques, means for videotaping and recording a user's sales performance, means for transmitting the videotaped and recorded data to the server, means for the server to analyze the videotaped and recorded data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for the server to use a deep learning model for the analysis, means for the user terminal to upload collected performance data to the server, and means for the user to receive feedback in real time via the terminal. This allows users to utilize advanced technology to constantly analyze their sales performance and receive instant feedback, enabling efficient self-learning and skill improvement.

[0686] "Data on sales techniques" refers to information stored in a database that includes information on sales activity scenarios, techniques, methods, facial expressions, tone and volume of voice, etc.

[0687] "User's sales performance" refers to the records of the user's actual sales presentations and interactions with customers.

[0688] "Video and audio data" refers to video and audio data recorded using a camera or microphone of a user's sales performance.

[0689] "Server" is a central processing unit that stores, analyzes, and generates feedback on sales methodology data.

[0690] A "deep learning model" is a type of artificial intelligence, specifically a technology that uses deep neural networks to analyze data and recognize patterns.

[0691] "Feedback" is information that provides an evaluation of the user's sales performance and areas for improvement based on the results of analysis by the server.

[0692] "User terminal" means a device that allows a user to record and record sales performance and receive feedback.

[0693] A "prompt sentence" is an instruction sentence that prompts the user to take a specific action or input.

[0694] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is implemented by the following procedure and configuration.

[0695] System configuration and hardware / software

[0696] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0697] Users can use the device to record and record their sales performance. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later.

[0698] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0699] The server analyzes the received performance data. Here, advanced analysis techniques such as deep learning models (specifically TensorFlow and PyTorch) are used to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[0700] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0701] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0702] Specific examples

[0703] For example, consider a case where a user performs during a business negotiation using a "serious expression" and "calm tone." In this case, the server's deep learning model analyzes the performance and provides specific feedback such as "You should smile more." If the tone of voice is appropriate, the system will give a high rating of "Voice tone is very good." This allows the user to clearly understand their strengths and areas for improvement and use this knowledge in their next performance.

[0704] Prompt Sentence Examples

[0705] Examples of prompts include:

[0706] "Please provide the analysis results of facial expressions and tone of voice at the progress stage of a sales negotiation, along with specific advice. Please give us feedback when users use a serious facial expression and a calm tone of voice."

[0707] In this way, the system of the present invention supports efficient learning of sales techniques and the leveling of skills, and is a powerful tool for users to improve themselves.

[0708] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0709] Step 1:

[0710] The server builds a database of sales techniques. The input includes success stories and model scenarios related to sales activities. The server analyzes this information and saves detailed information such as sales scenarios, facial expressions, and tone of voice in the database. Specifically, the sales scenarios include "initial greetings with customers" and "progress of sales negotiations." The output is model sales scenarios saved in the database.

[0711] Step 2:

[0712] Users use a device to record and record their sales performance. Input includes presentations and conversations with customers that the user makes using the device's camera and microphone. The device records and saves these performances as data. As a specific example, consider the case where a user simulates a sales negotiation. The output is the video and audio data that is saved on the device.

[0713] Step 3:

[0714] After the recording is complete, the device sends the collected performance data to the server. The input includes the recording data. The device displays a "Send" button, and when the user presses this button, the data is uploaded to the server. The output is the recording data stored on the server and prepared for analysis.

[0715] Step 4:

[0716] The server analyzes the received performance data using a deep learning model. Input includes video and audio data. The server uses the deep learning model to evaluate the user's facial expressions, tone of voice, etc. Specifically, it analyzes facial movements to identify "serious expressions" and "smiling faces." It also analyzes voice data to identify "calm tones" and "passionate tones." The output generates evaluation data of the user's sales techniques, facial expressions, and tone of voice.

[0717] Step 5:

[0718] The server generates feedback based on the analysis results. The input includes the analysis results. In addition to evaluating the sales technique, the server generates feedback that includes suggestions for improvement in facial expressions and tone of voice. Specific feedback might include advice such as, "Your facial expression seems a little stiff, so next time try to smile more relaxedly." The output is feedback data.

[0719] Step 6:

[0720] The generated feedback is sent from the server to the user terminal. The input includes feedback data. The server sends the feedback to the user terminal, and the user can receive the feedback in real time. The output is displayed on the user terminal.

[0721] Step 7:

[0722] The user checks the feedback on the device and uses it as a reference for improving their sales skills. The input includes the feedback. The user reviews the feedback and creates a specific action plan to help their next sales performance. Specifically, a goal might be set, such as "In the next presentation, I will speak in a brighter tone and smile more." The output is a specific action plan to improve the user's skills.

[0723] (Application example 1)

[0724] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0725] Conventional sales technique learning systems often required users to rely on self-evaluation, making it difficult to receive objective feedback. Furthermore, face-to-face instruction was required, which often meant location and time constraints. Furthermore, it was difficult to evaluate small details such as facial expressions and tone of voice, making it difficult to standardize skills. This resulted in issues such as the time it took for new employees to become immediately productive, delaying the improvement of the skills of the entire sales department.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0727] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for the server to generate feedback based on the analysis results, means for transmitting the feedback to a user terminal and for the user to confirm, and means for providing training scenarios using prompt sentences. This allows users to receive objective and detailed feedback and improve their sales skills efficiently without being restricted by location or time.

[0728] The "database for storing data on sales techniques" is an information infrastructure for systematically storing information related to sales performance, such as model sales scenarios, effective facial expressions, and appropriate tone and volume of voice.

[0729] "Means for recording video and audio of a user's sales performance" refers to a device or system for recording a user's sales activities using devices such as a camera and microphone.

[0730] "Means for transmitting video and audio data to a server" refers to a communication device or program for uploading video and audio data to a server via a network.

[0731] "Means for the server to analyze video and audio data and evaluate sales techniques, facial expressions, and tone of voice" refers to technology that allows the server to analyze video and audio data using deep learning models, etc., and evaluate sales techniques, the user's facial expressions, and tone of voice.

[0732] The "means for the server to generate feedback based on the analysis results" refers to a system or program for automatically generating feedback indicating areas for improvement and strengths to the user based on the analysis results.

[0733] The "means for transmitting feedback to the user terminal and for the user to confirm" refers to the communication technology and user interface for delivering the feedback generated by the server to the user's terminal and allowing the user to view it.

[0734] The "means for providing a training scenario using a prompt sentence" is a system that prepares sentences in advance to present a scenario for the user's training and displays them to the user.

[0735] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is actually operated with the cooperation of a server, terminals, and users.

[0736] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0737] Next, users can use a device to record and record their sales performance. Using a camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. Devices used for this purpose include smartphones, tablets, and smart glasses.

[0738] After the recording is complete, the device sends the collected performance data to the server. The recorded data is then uploaded to the server via the network. The communication protocol and data format are assumed to be the standard HTTP / S and JSON.

[0739] The server analyzes the received performance data. This analysis uses advanced analytical techniques such as deep learning models. For example, it builds models using libraries such as TensorFlow and PyTorch to evaluate the user's sales technique, facial expressions, and tone of voice. Specifically, it analyzes the user's facial movements to identify expressions such as a "serious expression" or a "smiling face," and analyzes voice data to identify expressions such as a "calm tone" or an "passionate tone."

[0740] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0741] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0742] For example, when a user starts a workout, a prompt is displayed, such as:

[0743] "Please handle the following scenarios with your customers:

[0744] 1. Handling customer complaints

[0745] 2. The customer is angry

[0746] 3. Maintain a calm tone of voice and a polite facial expression.

[0747] After displaying this prompt, the user records and films their response, and the data is sent to a server for analysis and feedback.

[0748] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0749] Step 1:

[0750] The server builds a database of sales techniques.

[0751] Input: Data such as model sales scenarios, effective facial expressions, appropriate voice tone and volume, etc.

[0752] Output: Sales technique information stored in the database.

[0753] How it works: The server collects new sales data and stores it in a database, where it is categorized, tagged, and stored in an easy-to-search format.

[0754] Step 2:

[0755] Users can use the device to record and record their sales performance.

[0756] Input: Camera and microphone to record sales performance.

[0757] Output: Recorded and audio data saved on the device.

[0758] How it works: The user uses the device's camera and microphone to record and record their sales activities. Once the recording is complete, the data is temporarily saved on the device.

[0759] Step 3:

[0760] The terminal transmits the collected performance data to the server.

[0761] Input: Video and audio data stored on the device.

[0762] Output: Performance data uploaded to the server.

[0763] Operation: The device sends video and audio data to the server via the network. The data is sent using the HTTP / S protocol. The server receives the data and stores it for analysis.

[0764] Step 4:

[0765] The server analyzes the video and audio data and evaluates sales techniques, facial expressions, and tone of voice.

[0766] Input: Video and audio data stored on the server.

[0767] Output: Evaluation results on sales techniques, facial expressions, and tone of voice.

[0768] How it works: The server analyzes the data using deep learning models (such as TensorFlow or PyTorch), using image analysis to analyze facial expressions and audio analysis to evaluate tone of voice.

[0769] Step 5:

[0770] The server generates feedback based on the analysis results.

[0771] Input: Evaluation results for sales technique, facial expression, and tone of voice.

[0772] Output: The generated feedback information.

[0773] How it works: Based on the analysis results, the server generates feedback that indicates the user's sales skills and areas for improvement. The feedback is designed to include specific advice and evaluation.

[0774] Step 6:

[0775] The server transmits the generated feedback to the user terminal.

[0776] Input: The generated feedback information.

[0777] Output: Feedback displayed on the user's terminal.

[0778] Operation: The server sends the generated feedback information to the user's terminal via the network, where the user can view the feedback on the terminal.

[0779] Step 7:

[0780] Provide training scenarios using prompts.

[0781] Input: prompt, training scenario.

[0782] Output: The training scenario presented to the user.

[0783] Operation: The server selects a training scenario from the database and sends it to the user's device as a prompt. The user then follows the prompt to train. "Please handle the following customer situations: 1. Handling a customer complaint. 2. The customer is angry. 3. Maintain a calm tone of voice and a polite facial expression."

[0784] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0785] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[0786] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0787] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0788] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0789] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[0790] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0791] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0792] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0793] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[0794] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[0795] The processing flow will be explained below.

[0796] Step 1:

[0797] The server creates a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tones and volumes. For example, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0798] Step 2:

[0799] Users can record their sales performance using the device. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0800] Step 3:

[0801] The device transmits the recorded performance data to a server, where it is stored and prepared for analysis using a data upload function via the internet.

[0802] Step 4:

[0803] The server analyzes the received data, using highly accurate analytical methods such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling smile." It also analyzes audio data to identify whether a person has a "calm tone" or an "passionate tone."

[0804] Step 5:

[0805] The server then uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it analyzes changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0806] Step 6:

[0807] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0808] Step 7:

[0809] The server sends the generated feedback to the terminal, which provides the feedback in a format that is easy for the user to view and can check it on the terminal.

[0810] Step 8:

[0811] Users can check the feedback on their devices and use it as a reference for improving their sales skills. This allows users to understand areas for improvement based on the feedback and continue to improve themselves.

[0812] Step 9:

[0813] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve their skills. By repeating this process, users can hone their sales techniques and put them to practical use.

[0814] Example 2

[0815] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0816] There is a need for a system that can efficiently learn and standardize sales techniques and provide advanced feedback that takes users' emotions into account. However, existing systems have limited ability to evaluate users' sales performance, and in particular lack feedback that takes users' emotions into account. This makes it difficult for users to recognize fluctuations in their performance due to their own emotions and use them to improve.

[0817] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0818] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for using an emotion engine to analyze the user's emotions, and means for reflecting the emotion analysis results in the generated feedback. This allows the user to receive feedback that takes into account both their sales performance and emotions, enabling more effective learning and skill improvement.

[0819] "Sales technique data" refers to data that includes detailed information such as specific scenarios related to sales processes and customer interactions, facial expressions, tone of voice, and volume.

[0820] A "database" is a collection of data related to sales techniques that is structured and stored so that it can be easily searched and referenced.

[0821] "User" means an individual or organization that intends to use this system to improve their own sales performance.

[0822] "Means for recording and recording sales performance" includes cameras and microphones for recording the user's sales activities as video and audio, as well as devices and programs for controlling their operation.

[0823] "Video and audio data" refers to video and audio data that records the user's business activities.

[0824] A "server" is a computer system that performs central processing such as storing data, analyzing data, and generating feedback.

[0825] A "deep learning model" is a type of artificial intelligence that uses neural networks to learn patterns from large amounts of data and perform analysis and classification.

[0826] The "emotion engine" is software that analyzes and recognizes a user's emotional state based on their video and audio data.

[0827] "Feedback" refers to evaluations and suggestions for improvement provided based on the analysis results of the sales performance recorded by the user.

[0828] "User terminal" means a device used by a user to record video and audio, and to receive feedback from the server.

[0829] "Analysis results" refer to information on evaluations and emotional states obtained from analysis by the video and audio data and the emotion engine.

[0830] "Means for evaluating sales techniques, facial expressions, and tone of voice" refers to the process in which the server analyzes video and audio data using deep learning models, etc., and evaluates the sales techniques, facial expressions, and tone of voice as a result.

[0831] "Means for reflecting the emotion analysis results in the generated feedback" refers to the process of creating detailed feedback that includes the user's emotional state based on the analysis results from the emotion engine.

[0832] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[0833] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[0834] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[0835] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0836] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[0837] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[0838] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[0839] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0840] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[0841] Example prompt: "We've recorded your performance in reaching new customers. We'd like you to analyze this data and provide feedback on how we can improve."

[0842] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[0843] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0844] Step 1:

[0845] The server builds a sales technique database.

[0846] Input: Information related to sales techniques such as sales scenarios, facial expression data, and tone of voice data.

[0847] Data processing: The information related to these sales techniques is stored in a database in a structured format.

[0848] Output: A database of sales techniques.

[0849] Specific operation: The server collects sales scenarios, facial expression data, and voice tone data, and stores them in a database. For example, it stores a scenario called "Approaching a new customer."

[0850] Step 2:

[0851] Users can record and record their sales performance.

[0852] Input: User's actual sales performance (video and audio).

[0853] Data processing: The user's sales performance is recorded using a camera and microphone.

[0854] Output: Video and audio recording of performance data.

[0855] Specific operation: The user presses the record button on the device and performs a mock sales scenario. The prompt sentence is "Please record and record the mock sales scenario."

[0856] Step 3:

[0857] The terminal transmits the collected performance data to the server.

[0858] Input: Video and audio recording of performance data.

[0859] Data processing: After recording and audio recording, the data file is uploaded to the server via the network.

[0860] Output: Performance data sent to the server.

[0861] Specific operation: After recording and recording is completed, the data is transferred to the server and a message is displayed saying "Recording is complete. Data will be sent to the server."

[0862] Step 4:

[0863] The server analyzes the data.

[0864] Input: Transmitted performance data (video and audio).

[0865] Data processing: Deep learning models are used to analyze performance data and evaluate sales techniques, facial expressions, and tone of voice.

[0866] Output: Evaluation data on sales techniques, facial expressions, and tone of voice.

[0867] Specific operations: Analyzes facial movements from video data to identify "smiling faces," etc. Analyzes audio data to identify "calm tones."

[0868] Step 5:

[0869] The server analyzes emotions using an emotion engine.

[0870] Input: Performance data (video and audio).

[0871] Data processing: Analyze user emotions from both video and audio, and reflect the results in the evaluation of sales performance.

[0872] Output: User sentiment analysis results.

[0873] Specific operation: Analyzes changes in the user's facial expression and voice to determine emotions such as tension or relaxation. Displays the message "Analyzing user's facial expression and tone of voice."

[0874] Step 6:

[0875] The server generates the feedback.

[0876] Input: Evaluation data of sales techniques, facial expressions, and tone of voice, as well as emotion analysis results.

[0877] Data processing: Generate feedback based on the evaluation data and also reflect the results of user sentiment analysis.

[0878] Output: The generated feedback.

[0879] Specific actions: Generate specific advice such as, "Your tone of voice is good, but you should try to look a little more relaxed with your facial expression."

[0880] Step 7:

[0881] The server transmits the generated feedback to the user terminal.

[0882] Input: The generated feedback.

[0883] Data processing: Send the feedback to the user's device and notify the user.

[0884] Output: Feedback displayed on the user's device.

[0885] What happens: Sends feedback to the device and notifies the user, "You have new feedback. Check it out."

[0886] (Application example 2)

[0887] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0888] In the case of conventional sales-based efficient learning systems, it is difficult to properly recognize users' emotions and reflect them in feedback, making it difficult for users to effectively improve their self-study and customer service skills.In addition, the lack of specific learning support for stores and customer service work makes it difficult for new employees in particular to immediately contribute.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for building a database for storing data related to sales techniques, means for recording the user's sales performance, and means for transmitting the recorded data to the server. This makes it possible to comprehensively evaluate the user's sales techniques, facial expressions, tone of voice, and emotions, effectively supporting the learning of customer service skills in stores and customer service work. Furthermore, by providing specific feedback using the generative AI model, the user can clearly understand their strengths and areas for improvement, promoting continuous skill improvement.

[0890] "Sales techniques" are a series of processes and techniques from the first contact with a customer to the conclusion of a contract, and include effective facial expressions, tone of voice, scenarios, etc.

[0891] "Performance" refers to the specific actions and behaviors that users actually perform in sales activities and customer interactions.

[0892] "Video recording" refers to the use of a camera or microphone to record a user's voice, facial expressions, movements, etc. as video and audio data.

[0893] A "server" is a computer system for analyzing stored data and generating feedback.

[0894] "Analysis" is the process of analyzing video and audio data to evaluate the user's sales techniques and emotions.

[0895] "Feedback" refers to evaluations and suggestions or advice on areas for improvement provided to users based on the analysis results.

[0896] "User terminal" refers to a device such as a smartphone or computer used by a user to check feedback.

[0897] "Emotion recognition" is a technology that identifies a user's emotions from video and audio data, and determines their state of tension or relaxation.

[0898] A "generative AI model" is a computer model that uses artificial intelligence to automatically generate text and data.

[0899] "Store and customer service operations" refers to all service activities, including sales activities and customer support, in physical stores.

[0900] The system of this invention combines efficient learning of sales techniques with user emotion recognition to help improve customer service skills in stores and customer service work. This system consists of the following components and processing steps.

[0901] Hardware and software used

[0902] Device: Devices such as smartphones and head-mounted displays are used to record and record users' sales performance.

[0903] Server: A computer system responsible for storing data, analyzing it, and generating feedback. It utilizes cloud services such as AWS or Google Cloud.

[0904] Deep learning model: Libraries such as TensorFlow and PyTorch are used for data analysis.

[0905] Emotion engine: Uses OpenFace, Microsoft Azure Emotion API, etc. to recognize emotions from video and audio data.

[0906] Generative AI model: Uses generative AI techniques to generate feedback statements.

[0907] System Program Processing

[0908] 1. Data Collection:

[0909] The device uses a camera and microphone to record and record the user's sales performance.

[0910] Example: Salespeople use their smartphones to record themselves practicing customer interactions.

[0911] 2. Data transmission:

[0912] The terminal transmits the collected data to the server.

[0913] Example: Recorded data is uploaded to a cloud server via Wi-Fi.

[0914] 3. Data Analysis:

[0915] A deep learning model is used on the server to analyze the recorded and audio data.

[0916] An emotion engine is used to identify the user's emotion.

[0917] Example: The server analyzes the user's facial expressions and tone of voice from video data and identifies "calm tone" and "tense expression."

[0918] 4. Feedback Generation:

[0919] Based on the analysis results, a generative AI model is used to generate specific feedback.

[0920] Example prompt sentence:

[0921] Feed your voice data into a deep learning model to identify patterns such as "calm tone," "passionate tone," and "friendly tone."

[0922] Analyze facial video data and identify expressions such as "smiling," "serious," "relaxed," and "tense."

[0923] Identify emotions from video and audio data and identify the emotions of salespeople when interacting with customers.

[0924] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[0925] Specific feedback examples:

[0926] Your tone of voice is very calm and puts the customer at ease. However, your facial expression is a little stiff, so try to relax and smile a little more. Your overall performance is good, and you have a very good foundation for building trust with the customer.

[0927] 5. Send Feedback:

[0928] The generated feedback is sent from the server to the user terminal, where the user can check it on the terminal.

[0929] Add specific examples to the description

[0930] For example, suppose a salesperson uses their smartphone to record a customer interaction practice. The recorded data is then sent to a server and analyzed using a deep learning model. Based on the analysis results, the generative AI model generates feedback such as "Your tone of voice is very good, but your facial expression is stiff," which is then sent to the user's device. The user can then review the feedback and use it in their next customer interaction.

[0931] This system allows users to clearly understand their strengths and areas for improvement, enabling continuous skill improvement. It also allows new employees, in particular, to quickly acquire customer service skills and become immediately effective.

[0932] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0933] Step 1: Data collection

[0934] Using a device (smartphone or head-mounted display), the user can record and record their sales performance. Specifically, a camera and microphone are used to record customer service practice and actual conversations. The input data is video and audio data, and the output is a video and audio file.

[0935] Step 2: Send data

[0936] The device sends the video and audio data collected in step 1 to the server. Specifically, it uploads the data using an HTTP request or WebSocket protocol over Wi-Fi or a mobile data network. The input is the video and audio file, and the output is the data stored on the server.

[0937] Step 3: Data analysis

[0938] The server analyzes the video and audio data it receives. In this step, a deep learning model is used to analyze facial expressions from the video data and tone of voice from the audio data. The input is the video and audio data, and the output is the analysis results: facial expression data, tone of voice data, and emotion recognition results. Specifically, a model using TensorFlow and PyTorch is run to extract features such as "calm tone," "serious expression," and "tension."

[0939] Step 4: Emotion Recognition

[0940] The server recognizes the user's emotions using an emotion engine (e.g., OpenFace or Microsoft Azure Emotion API) based on the analysis results from step 3. The input is facial expression data and tone of voice data, and the output is the user's emotional assessment (e.g., "relaxed" or "tense").

[0941] Step 5: Feedback generation

[0942] The server uses a generative AI model to generate specific feedback sentences based on the analysis results and emotion recognition results. Using example prompt sentences, feedback including evaluation points and areas for improvement is generated. The input is the analysis results and emotion evaluation data, and the output is the feedback sentence. Specifically, the following prompt sentences are used:

[0943] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[0944] Step 6: Send your feedback

[0945] The server sends the generated feedback sentence to the user device. In this step, a cloud messaging service (e.g., Firebase Cloud Messaging) is used to display the feedback sentence on the user device. The input is the feedback sentence, and the output is the feedback displayed on the user device.

[0946] Step 7: Check feedback and try again

[0947] The user checks the feedback on the device and attempts to perform the sales performance again based on the areas for improvement. Specifically, the user understands the feedback and records new conversations and customer responses. The input is the feedback text, and the output is video and audio data of the improved sales performance.

[0948] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0949] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0950] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0951] [Fourth embodiment]

[0952] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0953] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0954] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0955] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0956] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0957] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0958] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0959] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0960] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0961] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0962] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0963] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0964] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0965] The present invention is a system for efficiently learning sales techniques and leveling skills, and is implemented by the following procedure and configuration.

[0966] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[0967] Next, users can use the device to record and record their sales performance. Users can use the camera and microphone to record presentations and conversations with customers, allowing them to review their sales techniques later.

[0968] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[0969] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[0970] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[0971] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[0972] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, the user will receive feedback such as "You should smile more." If the tone of voice is appropriate, the user will be evaluated as "Very good tone of voice." This allows the user to clearly understand their strengths and areas for improvement and use this information in their next performance.

[0973] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and provides a powerful tool for users to improve themselves.

[0974] The processing flow will be explained below.

[0975] Step 1:

[0976] The server creates a database of sales techniques. Detailed information such as model sales scenarios, facial expressions, tone and volume of voice is entered into this database. For example, scenarios such as "initial greeting with a customer," "progression of a negotiation," and "successful closing" are saved.

[0977] Step 2:

[0978] Users can use their devices to record and record their sales performance. Using a camera and microphone, users can record their speech and facial expressions for later analysis. For example, they can conduct a mock sales meeting in front of the camera.

[0979] Step 3:

[0980] The device sends the recorded performance data to a server, where it is securely uploaded and stored. This is done using the data upload function via the internet.

[0981] Step 4:

[0982] The server analyzes the received data, using highly accurate analytical techniques such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling face."

[0983] Step 5:

[0984] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique and specific points for improvement such as facial expressions and tone of voice. For example, the server may provide advice such as, "Your tone of voice is good, but you should smile a bit more."

[0985] Step 6:

[0986] The server sends the generated feedback to the terminal, where it is displayed for the user to view. The feedback is provided in a format that is easy for the user to view.

[0987] Step 7:

[0988] Users check the feedback on their device. Based on the feedback provided, they can understand the strengths and areas for improvement in their sales performance. For example, if they are evaluated as having an "appropriate tone of voice" and are told that their "facial expression is stiff," they can use that improvement in their next training session.

[0989] Step 8:

[0990] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve themselves and their skills. Users can repeat this process to hone their sales techniques.

[0991] The above are the detailed processing steps in the system of the present invention, which allows users to efficiently learn sales techniques and improve themselves.

[0992] Example 1

[0993] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0994] There is a need for a more efficient and standardized way to learn sales techniques and improve skills. However, current systems lack effective feedback and provide users with no easy way to evaluate and improve their performance. Furthermore, they lack advanced technology for analyzing video and audio data, resulting in limited quality of feedback.

[0995] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0996] In this invention, the server includes means for building a database for storing data on sales techniques, means for videotaping and recording a user's sales performance, means for transmitting the videotaped and recorded data to the server, means for the server to analyze the videotaped and recorded data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for the server to use a deep learning model for the analysis, means for the user terminal to upload collected performance data to the server, and means for the user to receive feedback in real time via the terminal. This allows users to utilize advanced technology to constantly analyze their sales performance and receive instant feedback, enabling efficient self-learning and skill improvement.

[0997] "Data on sales techniques" refers to information stored in a database that includes information on sales activity scenarios, techniques, methods, facial expressions, tone and volume of voice, etc.

[0998] "User's sales performance" refers to the records of the user's actual sales presentations and interactions with customers.

[0999] "Video and audio data" refers to video and audio data recorded using a camera or microphone of a user's sales performance.

[1000] "Server" is a central processing unit that stores, analyzes, and generates feedback on sales methodology data.

[1001] A "deep learning model" is a type of artificial intelligence, specifically a technology that uses deep neural networks to analyze data and recognize patterns.

[1002] "Feedback" is information that provides an evaluation of the user's sales performance and areas for improvement based on the results of analysis by the server.

[1003] "User terminal" means a device that allows a user to record and record sales performance and receive feedback.

[1004] A "prompt sentence" is an instruction sentence that prompts the user to take a specific action or input.

[1005] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is implemented by the following procedure and configuration.

[1006] System configuration and hardware / software

[1007] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[1008] Users can use the device to record and record their sales performance. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later.

[1009] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[1010] The server analyzes the received performance data. Here, advanced analysis techniques such as deep learning models (specifically TensorFlow and PyTorch) are used to evaluate the user's sales technique, facial expressions, and tone of voice. For example, a deep learning model analyzes the user's facial movements to identify whether they have a "serious expression" or a "smiling face." It also analyzes voice data to identify whether they have a "calm tone" or an "enthusiastic tone."

[1011] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[1012] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[1013] Specific examples

[1014] For example, consider a case where a user performs during a business negotiation using a "serious expression" and "calm tone." In this case, the server's deep learning model analyzes the performance and provides specific feedback such as "You should smile more." If the tone of voice is appropriate, the system will give a high rating of "Voice tone is very good." This allows the user to clearly understand their strengths and areas for improvement and use this knowledge in their next performance.

[1015] Prompt Sentence Examples

[1016] Examples of prompts include:

[1017] "Please provide the analysis results of facial expressions and tone of voice at the progress stage of a sales negotiation, along with specific advice. Please give us feedback when users use a serious facial expression and a calm tone of voice."

[1018] In this way, the system of the present invention supports efficient learning of sales techniques and the leveling of skills, and is a powerful tool for users to improve themselves.

[1019] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1020] Step 1:

[1021] The server builds a database of sales techniques. The input includes success stories and model scenarios related to sales activities. The server analyzes this information and saves detailed information such as sales scenarios, facial expressions, and tone of voice in the database. Specifically, the sales scenarios include "initial greetings with customers" and "progress of sales negotiations." The output is model sales scenarios saved in the database.

[1022] Step 2:

[1023] Users use a device to record and record their sales performance. Input includes presentations and conversations with customers that the user makes using the device's camera and microphone. The device records and saves these performances as data. As a specific example, consider the case where a user simulates a sales negotiation. The output is the video and audio data that is saved on the device.

[1024] Step 3:

[1025] After the recording is complete, the device sends the collected performance data to the server. The input includes the recording data. The device displays a "Send" button, and when the user presses this button, the data is uploaded to the server. The output is the recording data stored on the server and prepared for analysis.

[1026] Step 4:

[1027] The server analyzes the received performance data using a deep learning model. Input includes video and audio data. The server uses the deep learning model to evaluate the user's facial expressions, tone of voice, etc. Specifically, it analyzes facial movements to identify "serious expressions" and "smiling faces." It also analyzes voice data to identify "calm tones" and "passionate tones." The output generates evaluation data of the user's sales techniques, facial expressions, and tone of voice.

[1028] Step 5:

[1029] The server generates feedback based on the analysis results. The input includes the analysis results. In addition to evaluating the sales technique, the server generates feedback that includes suggestions for improvement in facial expressions and tone of voice. Specific feedback might include advice such as, "Your facial expression seems a little stiff, so next time try to smile more relaxedly." The output is feedback data.

[1030] Step 6:

[1031] The generated feedback is sent from the server to the user terminal. The input includes feedback data. The server sends the feedback to the user terminal, and the user can receive the feedback in real time. The output is displayed on the user terminal.

[1032] Step 7:

[1033] The user checks the feedback on the device and uses it as a reference for improving their sales skills. The input includes the feedback. The user reviews the feedback and creates a specific action plan to help their next sales performance. Specifically, a goal might be set, such as "In the next presentation, I will speak in a brighter tone and smile more." The output is a specific action plan to improve the user's skills.

[1034] (Application example 1)

[1035] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1036] Conventional sales technique learning systems often required users to rely on self-evaluation, making it difficult to receive objective feedback. Furthermore, face-to-face instruction was required, which often meant location and time constraints. Furthermore, it was difficult to evaluate small details such as facial expressions and tone of voice, making it difficult to standardize skills. This resulted in issues such as the time it took for new employees to become immediately productive, delaying the improvement of the skills of the entire sales department.

[1037] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1038] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for the server to generate feedback based on the analysis results, means for transmitting the feedback to a user terminal and for the user to confirm, and means for providing training scenarios using prompt sentences. This allows users to receive objective and detailed feedback and improve their sales skills efficiently without being restricted by location or time.

[1039] The "database for storing data on sales techniques" is an information infrastructure for systematically storing information related to sales performance, such as model sales scenarios, effective facial expressions, and appropriate tone and volume of voice.

[1040] "Means for recording video and audio of a user's sales performance" refers to a device or system for recording a user's sales activities using devices such as a camera and microphone.

[1041] "Means for transmitting video and audio data to a server" refers to a communication device or program for uploading video and audio data to a server via a network.

[1042] "Means for the server to analyze video and audio data and evaluate sales techniques, facial expressions, and tone of voice" refers to technology that allows the server to analyze video and audio data using deep learning models, etc., and evaluate sales techniques, the user's facial expressions, and tone of voice.

[1043] The "means for the server to generate feedback based on the analysis results" refers to a system or program for automatically generating feedback indicating areas for improvement and strengths to the user based on the analysis results.

[1044] The "means for transmitting feedback to the user terminal and for the user to confirm" refers to the communication technology and user interface for delivering the feedback generated by the server to the user's terminal and allowing the user to view it.

[1045] The "means for providing a training scenario using a prompt sentence" is a system that prepares sentences in advance to present a scenario for the user's training and displays them to the user.

[1046] The present invention is a system for efficiently learning sales techniques and leveling skills. This system is actually operated with the cooperation of a server, terminals, and users.

[1047] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. For example, specific scenarios such as "initial greeting with a customer," "progressing a deal," and "closing" are saved.

[1048] Next, users can use a device to record and record their sales performance. Using a camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. Devices used for this purpose include smartphones, tablets, and smart glasses.

[1049] After the recording is complete, the device sends the collected performance data to the server. The recorded data is then uploaded to the server via the network. The communication protocol and data format are assumed to be the standard HTTP / S and JSON.

[1050] The server analyzes the received performance data. This analysis uses advanced analytical techniques such as deep learning models. For example, it builds models using libraries such as TensorFlow and PyTorch to evaluate the user's sales technique, facial expressions, and tone of voice. Specifically, it analyzes the user's facial movements to identify expressions such as a "serious expression" or a "smiling face," and analyzes voice data to identify expressions such as a "calm tone" or an "passionate tone."

[1051] Based on the analysis results, the server generates feedback, which includes an evaluation of the user's sales technique as well as suggestions for improving facial expressions and tone of voice. For example, the server may provide specific advice such as, "Your facial expression seems a little stiff. Next time, try to smile more relaxedly."

[1052] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[1053] For example, when a user starts a workout, a prompt is displayed, such as:

[1054] "Please handle the following scenarios with your customers:

[1055] 1. Handling customer complaints

[1056] 2. The customer is angry

[1057] 3. Maintain a calm tone of voice and a polite facial expression.

[1058] After displaying this prompt, the user records and films their response, and the data is sent to a server for analysis and feedback.

[1059] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1060] Step 1:

[1061] The server builds a database of sales techniques.

[1062] Input: Data such as model sales scenarios, effective facial expressions, appropriate voice tone and volume, etc.

[1063] Output: Sales technique information stored in the database.

[1064] How it works: The server collects new sales data and stores it in a database, where it is categorized, tagged, and stored in an easy-to-search format.

[1065] Step 2:

[1066] Users can use the device to record and record their sales performance.

[1067] Input: Camera and microphone to record sales performance.

[1068] Output: Recorded and audio data saved on the device.

[1069] How it works: The user uses the device's camera and microphone to record and record their sales activities. Once the recording is complete, the data is temporarily saved on the device.

[1070] Step 3:

[1071] The terminal transmits the collected performance data to the server.

[1072] Input: Video and audio data stored on the device.

[1073] Output: Performance data uploaded to the server.

[1074] Operation: The device sends video and audio data to the server via the network. The data is sent using the HTTP / S protocol. The server receives the data and stores it for analysis.

[1075] Step 4:

[1076] The server analyzes the video and audio data and evaluates sales techniques, facial expressions, and tone of voice.

[1077] Input: Video and audio data stored on the server.

[1078] Output: Evaluation results on sales techniques, facial expressions, and tone of voice.

[1079] How it works: The server analyzes the data using deep learning models (such as TensorFlow or PyTorch), using image analysis to analyze facial expressions and audio analysis to evaluate tone of voice.

[1080] Step 5:

[1081] The server generates feedback based on the analysis results.

[1082] Input: Evaluation results for sales technique, facial expression, and tone of voice.

[1083] Output: The generated feedback information.

[1084] How it works: Based on the analysis results, the server generates feedback on the user's sales skills and areas for improvement. The feedback is designed to include specific advice and evaluation.

[1085] Step 6:

[1086] The server transmits the generated feedback to the user terminal.

[1087] Input: The generated feedback information.

[1088] Output: Feedback displayed on the user's terminal.

[1089] Operation: The server sends the generated feedback information to the user's terminal via the network, where the user can view the feedback on the terminal.

[1090] Step 7:

[1091] Provide training scenarios using prompts.

[1092] Input: prompt, training scenario.

[1093] Output: The training scenario presented to the user.

[1094] Operation: The server selects a training scenario from the database and sends it to the user's device as a prompt. The user then follows the prompt to train. "Please handle the following customer situations: 1. Handling a customer complaint. 2. The customer is angry. 3. Maintain a calm tone of voice and a polite facial expression."

[1095] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1096] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[1097] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[1098] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[1099] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[1100] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[1101] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[1102] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[1103] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[1104] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[1105] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[1106] The processing flow will be explained below.

[1107] Step 1:

[1108] The server creates a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tones and volumes. For example, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[1109] Step 2:

[1110] Users can record their sales performance using the device. They can use the camera and microphone to record presentations and conversations with customers, and review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[1111] Step 3:

[1112] The device transmits the recorded performance data to a server, where it is stored and prepared for analysis using a data upload function via the internet.

[1113] Step 4:

[1114] The server analyzes the received data, using highly accurate analytical methods such as deep learning models to identify and evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify whether a person has a "serious expression" or a "smiling smile." It also analyzes audio data to identify whether a person has a "calm tone" or an "passionate tone."

[1115] Step 5:

[1116] The server then uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it analyzes changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[1117] Step 6:

[1118] The server generates feedback based on the analysis results. The feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[1119] Step 7:

[1120] The server sends the generated feedback to the terminal, which provides the feedback in a format that is easy for the user to view and can check it on the terminal.

[1121] Step 8:

[1122] Users can check the feedback on their devices and use it as a reference for improving their sales skills. This allows users to understand areas for improvement based on the feedback and continue to improve themselves.

[1123] Step 9:

[1124] Based on the feedback, users can then record and record their sales performance again, allowing them to continually improve their skills. By repeating this process, users can hone their sales techniques and put them to practical use.

[1125] Example 2

[1126] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1127] There is a need for a system that can efficiently learn and standardize sales techniques and provide advanced feedback that takes users' emotions into account. However, existing systems have limited ability to evaluate users' sales performance, and in particular lack feedback that takes users' emotions into account. This makes it difficult for users to recognize fluctuations in their performance due to their own emotions and use them to improve.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1129] In this invention, the server includes means for constructing a database for storing data on sales techniques, means for recording video and audio of a user's sales performance, means for transmitting the video and audio data to the server, means for the server to analyze the video and audio data and evaluate the sales techniques, facial expressions, and tone of voice, means for generating feedback based on the analysis results, means for transmitting the feedback to the user terminal, means for using an emotion engine to analyze the user's emotions, and means for reflecting the emotion analysis results in the generated feedback. This allows the user to receive feedback that takes into account both their sales performance and emotions, enabling more effective learning and skill improvement.

[1130] "Sales technique data" refers to data that includes detailed information such as specific scenarios related to sales processes and customer interactions, facial expressions, tone of voice, and volume.

[1131] A "database" is a collection of data related to sales techniques that is structured and stored so that it can be easily searched and referenced.

[1132] "User" means an individual or organization that intends to use this system to improve their own sales performance.

[1133] "Means for recording and recording sales performance" includes cameras and microphones for recording the user's sales activities as video and audio, as well as devices and programs for controlling their operation.

[1134] "Video and audio data" refers to video and audio data that records the user's business activities.

[1135] A "server" is a computer system that performs central processing such as storing data, analyzing data, and generating feedback.

[1136] A "deep learning model" is a type of artificial intelligence that uses neural networks to learn patterns from large amounts of data and perform analysis and classification.

[1137] The "emotion engine" is software that analyzes and recognizes a user's emotional state based on their video and audio data.

[1138] "Feedback" refers to evaluations and suggestions for improvement provided based on the analysis results of the sales performance recorded by the user.

[1139] "User terminal" means a device used by a user to record video and audio, and to receive feedback from the server.

[1140] "Analysis results" refer to information on evaluations and emotional states obtained from analysis by the video and audio data and the emotion engine.

[1141] "Means for evaluating sales techniques, facial expressions, and tone of voice" refers to the process in which the server analyzes video and audio data using deep learning models, etc., and evaluates the sales techniques, facial expressions, and tone of voice as a result.

[1142] "Means for reflecting the emotion analysis results in the generated feedback" refers to the process of creating detailed feedback that includes the user's emotional state based on the analysis results from the emotion engine.

[1143] The present invention is a system that aims to efficiently learn sales techniques and standardize skills, and also combines an emotion engine that recognizes the user's emotions.

[1144] First, the server builds a database of sales techniques, including detailed information such as model sales scenarios, effective facial expressions, and appropriate voice tone and volume. Specifically, scenarios such as "initial greeting with a customer," "progressing a negotiation," and "closing" are saved.

[1145] Next, users can use the device to record and record their sales performance. Using the camera and microphone, users can record presentations and conversations with customers, allowing them to review their sales techniques later. For example, they can conduct mock sales negotiations in front of the camera.

[1146] After recording is complete, the device sends the collected performance data to a server where it is stored and prepared for analysis.

[1147] The server analyzes the received performance data. Here, it uses advanced analytical techniques such as deep learning models to evaluate the user's sales techniques, facial expressions, and tone of voice. For example, it analyzes facial movements from video data to identify "serious expressions" and "smiling faces." It also analyzes audio data to identify "calm tones" and "passionate tones."

[1148] Furthermore, the server uses an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from both video and audio, and reflects the results in the evaluation of sales performance. For example, it can analyze changes in the user's facial expressions and voice to determine whether the user is nervous or relaxed.

[1149] The server generates feedback based on the analysis results. This feedback includes an evaluation of the user's sales technique, suggestions for improvement in facial expressions and tone of voice, and even an evaluation of emotions using an emotion engine. For example, the server may provide advice such as, "Your tone of voice is good, but you should try to be more relaxed with your facial expressions."

[1150] The generated feedback is sent from the server to the user's device. The user can check the feedback on their device and use it as a reference for improving their sales skills. This allows users to improve themselves at any time, regardless of location or time. It also enables new employees to learn effectively and become immediately effective.

[1151] For example, suppose a user performs a business meeting using a "serious expression" and "calm tone." In this case, if the emotion engine recognizes the user's emotion as "nervous," it will provide specific feedback such as, "You appear calm, but your expression is stiff. Try to smile more relaxedly." It will also include an evaluation such as, "Your tone of voice is very good." This allows the user to clearly understand their strengths and areas for improvement, and use this information in their next performance.

[1152] Example prompt: "We've recorded your performance in reaching new customers. We'd like you to analyze this data and provide feedback on how we can improve."

[1153] As described above, the system of the present invention supports efficient learning of sales techniques and skill leveling, and by combining it with an emotion engine, it provides a powerful tool for users to improve themselves.

[1154] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1155] Step 1:

[1156] The server builds a sales technique database.

[1157] Input: Information related to sales techniques such as sales scenarios, facial expression data, and tone of voice data.

[1158] Data processing: The information related to these sales techniques is stored in a database in a structured format.

[1159] Output: A database of sales techniques.

[1160] Specific operation: The server collects sales scenarios, facial expression data, and voice tone data, and stores them in a database. For example, it stores a scenario called "Approaching a new customer."

[1161] Step 2:

[1162] Users can record and record their sales performance.

[1163] Input: User's actual sales performance (video and audio).

[1164] Data processing: The user's sales performance is recorded using a camera and microphone.

[1165] Output: Video and audio recording of performance data.

[1166] Specific operation: The user presses the record button on the device and performs a mock sales scenario. The prompt sentence is "Please record and record the mock sales scenario."

[1167] Step 3:

[1168] The terminal transmits the collected performance data to the server.

[1169] Input: Video and audio recording of performance data.

[1170] Data processing: After recording and audio recording, the data file is uploaded to the server via the network.

[1171] Output: Performance data sent to the server.

[1172] Specific operation: After recording and recording is completed, the data is transferred to the server and a message is displayed saying "Recording is complete. Data will be sent to the server."

[1173] Step 4:

[1174] The server analyzes the data.

[1175] Input: Transmitted performance data (video and audio).

[1176] Data processing: Deep learning models are used to analyze performance data and evaluate sales techniques, facial expressions, and tone of voice.

[1177] Output: Evaluation data on sales techniques, facial expressions, and tone of voice.

[1178] Specific operations: Analyzes facial movements from video data to identify "smiling faces," etc. Analyzes audio data to identify "calm tones."

[1179] Step 5:

[1180] The server analyzes emotions using an emotion engine.

[1181] Input: Performance data (video and audio).

[1182] Data processing: Analyze user emotions from both video and audio, and reflect the results in the evaluation of sales performance.

[1183] Output: User sentiment analysis results.

[1184] Specific operation: Analyzes changes in the user's facial expression and voice to determine emotions such as tension or relaxation. Displays the message "Analyzing user's facial expression and tone of voice."

[1185] Step 6:

[1186] The server generates the feedback.

[1187] Input: Evaluation data of sales techniques, facial expressions, and tone of voice, as well as emotion analysis results.

[1188] Data processing: Generate feedback based on the evaluation data and also reflect the results of user sentiment analysis.

[1189] Output: The generated feedback.

[1190] Specific actions: Generate specific advice such as, "Your tone of voice is good, but you should try to look a little more relaxed with your facial expression."

[1191] Step 7:

[1192] The server transmits the generated feedback to the user terminal.

[1193] Input: The generated feedback.

[1194] Data processing: Send the feedback to the user's device and notify the user.

[1195] Output: Feedback displayed on the user's device.

[1196] What happens: Sends feedback to the device and notifies the user, "You have new feedback. Check it out."

[1197] (Application example 2)

[1198] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1199] In the case of conventional sales-based efficient learning systems, it is difficult to properly recognize users' emotions and reflect them in feedback, making it difficult for users to effectively improve their self-study and customer service skills.In addition, the lack of specific learning support for stores and customer service work makes it difficult for new employees in particular to immediately contribute.

[1200] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for building a database for storing data related to sales techniques, means for recording the user's sales performance, and means for transmitting the recorded data to the server. This makes it possible to comprehensively evaluate the user's sales techniques, facial expressions, tone of voice, and emotions, effectively supporting the learning of customer service skills in stores and customer service work. Furthermore, by providing specific feedback using the generative AI model, the user can clearly understand their strengths and areas for improvement, promoting continuous skill improvement.

[1201] "Sales techniques" are a series of processes and techniques from the first contact with a customer to the conclusion of a contract, and include effective facial expressions, tone of voice, scenarios, etc.

[1202] "Performance" refers to the specific actions and behaviors that users actually perform in sales activities and customer interactions.

[1203] "Video recording" refers to the use of a camera or microphone to record a user's voice, facial expressions, movements, etc. as video and audio data.

[1204] A "server" is a computer system for analyzing stored data and generating feedback.

[1205] "Analysis" is the process of analyzing video and audio data to evaluate the user's sales techniques and emotions.

[1206] "Feedback" refers to evaluations and suggestions or advice on areas for improvement provided to users based on the analysis results.

[1207] "User terminal" refers to a device such as a smartphone or computer used by a user to check feedback.

[1208] "Emotion recognition" is a technology that identifies a user's emotions from video and audio data, and determines their state of tension or relaxation.

[1209] A "generative AI model" is a computer model that uses artificial intelligence to automatically generate text and data.

[1210] "Store and customer service operations" refers to all service activities, including sales activities and customer support, in physical stores.

[1211] The system of this invention combines efficient learning of sales techniques with user emotion recognition to help improve customer service skills in stores and customer service work. This system consists of the following components and processing steps.

[1212] Hardware and software used

[1213] Device: Devices such as smartphones and head-mounted displays are used to record and record users' sales performance.

[1214] Server: A computer system responsible for storing data, analyzing it, and generating feedback. It utilizes cloud services such as AWS or Google Cloud.

[1215] Deep learning model: Libraries such as TensorFlow and PyTorch are used for data analysis.

[1216] Emotion engine: Uses OpenFace, Microsoft Azure Emotion API, etc. to recognize emotions from video and audio data.

[1217] Generative AI model: Uses generative AI techniques to generate feedback statements.

[1218] System Program Processing

[1219] 1. Data Collection:

[1220] The device uses a camera and microphone to record and record the user's sales performance.

[1221] Example: Salespeople use their smartphones to record themselves practicing customer interactions.

[1222] 2. Data transmission:

[1223] The terminal transmits the collected data to the server.

[1224] Example: Recorded data is uploaded to a cloud server via Wi-Fi.

[1225] 3. Data Analysis:

[1226] A deep learning model is used on the server to analyze the recorded and audio data.

[1227] An emotion engine is used to identify the user's emotion.

[1228] Example: The server analyzes the user's facial expressions and tone of voice from video data and identifies "calm tone" and "tense expression."

[1229] 4. Feedback Generation:

[1230] Based on the analysis results, a generative AI model is used to generate specific feedback.

[1231] Example prompt sentence:

[1232] Feed your voice data into a deep learning model to identify patterns such as "calm tone," "passionate tone," and "friendly tone."

[1233] Analyze facial video data and identify expressions such as "smiling," "serious," "relaxed," and "tense."

[1234] Identify emotions from video and audio data and identify the emotions of salespeople when interacting with customers.

[1235] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[1236] Specific feedback examples:

[1237] Your tone of voice is very calm and puts the customer at ease. However, your facial expression is a little stiff, so try to relax and smile a little more. Your overall performance is good, and you have a very good foundation for building trust with the customer.

[1238] 5. Send Feedback:

[1239] The generated feedback is sent from the server to the user terminal, where the user can check it on the terminal.

[1240] Add specific examples to the description

[1241] For example, suppose a salesperson uses their smartphone to record a customer interaction practice. The recorded data is then sent to a server and analyzed using a deep learning model. Based on the analysis results, the generative AI model generates feedback such as "Your tone of voice is very good, but your facial expression is stiff," which is then sent to the user's device. The user can then review the feedback and use it in their next customer interaction.

[1242] This system allows users to clearly understand their strengths and areas for improvement, enabling continuous skill improvement. It also allows new employees, in particular, to quickly acquire customer service skills and become immediately effective.

[1243] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1244] Step 1: Data collection

[1245] Using a device (smartphone or head-mounted display), the user can record and record their sales performance. Specifically, a camera and microphone are used to record customer service practice and actual conversations. The input data is video and audio data, and the output is a video and audio file.

[1246] Step 2: Send data

[1247] The device sends the video and audio data collected in step 1 to the server. Specifically, it uploads the data using an HTTP request or WebSocket protocol over Wi-Fi or a mobile data network. The input is the video and audio file, and the output is the data stored on the server.

[1248] Step 3: Data analysis

[1249] The server analyzes the video and audio data it receives. In this step, a deep learning model is used to analyze facial expressions from the video data and tone of voice from the audio data. The input is the video and audio data, and the output is the analysis results: facial expression data, tone of voice data, and emotion recognition results. Specifically, a model using TensorFlow and PyTorch is run to extract features such as "calm tone," "serious expression," and "tension."

[1250] Step 4: Emotion Recognition

[1251] The server recognizes the user's emotions using an emotion engine (e.g., OpenFace or Microsoft Azure Emotion API) based on the analysis results from step 3. The input is facial expression data and tone of voice data, and the output is the user's emotional assessment (e.g., "relaxed" or "tense").

[1252] Step 5: Feedback generation

[1253] The server uses a generative AI model to generate specific feedback sentences based on the analysis results and emotion recognition results. Using example prompt sentences, feedback including evaluation points and areas for improvement is generated. The input is the analysis results and emotion evaluation data, and the output is the feedback sentence. Specifically, the following prompt sentences are used:

[1254] Generate feedback statements with specific areas for improvement and strengths based on your assessment of the salesperson's facial expressions, tone of voice, and emotions.

[1255] Step 6: Send your feedback

[1256] The server sends the generated feedback sentence to the user device. In this step, a cloud messaging service (e.g., Firebase Cloud Messaging) is used to display the feedback sentence on the user device. The input is the feedback sentence, and the output is the feedback displayed on the user device.

[1257] Step 7: Check feedback and try again

[1258] The user checks the feedback on the device and attempts to perform the sales performance again based on the areas for improvement. Specifically, the user understands the feedback and records new conversations and customer responses. The input is the feedback text, and the output is video and audio data of the improved sales performance.

[1259] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1260] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1261] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1262] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1263] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1264] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1265] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1266] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1267] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1268] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1269] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1270] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1271] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1272] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1273] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1274] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1275] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1276] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1277] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1278] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1279] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1280] The following is further disclosed regarding the above embodiment.

[1281] (Claim 1)

[1282] a means for constructing a database for storing data relating to sales techniques;

[1283] A means for recording and recording the user's sales performance;

[1284] A means for transmitting video and audio data to a server;

[1285] The server analyzes the video and audio data to evaluate sales techniques, facial expressions, and tone of voice.

[1286] means for generating feedback based on the analysis results;

[1287] means for transmitting feedback to a user terminal;

[1288] A system including:

[1289] (Claim 2)

[1290] 10. The system of claim 1, further comprising means for a user to view the feedback and reattempt the sales performance based on improvements.

[1291] (Claim 3)

[1292] The system described in claim 1, characterized in that a deep learning model is used to analyze video and audio data.

[1293] "Example 1"

[1294] (Claim 1)

[1295] a means for constructing a database for storing data relating to sales techniques;

[1296] A means for recording and recording the user's sales performance;

[1297] A means for transmitting video and audio data to a server;

[1298] The server analyzes the video and audio data to evaluate sales techniques, facial expressions, and tone of voice.

[1299] means for generating feedback based on the analysis results;

[1300] means for transmitting feedback to a user terminal;

[1301] A means by which the server uses deep learning models for analysis;

[1302] A means for uploading performance data collected by the user terminal to a server;

[1303] a means for users to receive real-time feedback via their devices;

[1304] A system including:

[1305] (Claim 2)

[1306] 10. The system of claim 1, further comprising means for a user to view the feedback and reattempt the sales performance based on improvements.

[1307] (Claim 3)

[1308] The system of claim 1, wherein a prompt sentence is used for detailed analysis of video and audio data.

[1309] "Application Example 1"

[1310] (Claim 1)

[1311] a means for constructing a database for storing data relating to sales techniques;

[1312] A means for recording and recording the user's sales performance;

[1313] A means for transmitting video and audio data to a server;

[1314] The server analyzes the video and audio data to evaluate sales techniques, facial expressions, and tone of voice.

[1315] a means for the server to generate feedback based on the analysis results;

[1316] means for transmitting the feedback to a user terminal and for the user to confirm;

[1317] a means for providing a training scenario using prompts;

[1318] A system including:

[1319] (Claim 2)

[1320] 10. The system of claim 1, further comprising means for a user to view the feedback and reattempt the sales performance based on improvements.

[1321] (Claim 3)

[1322] The system described in claim 1, characterized in that a deep learning model is used to analyze video and audio data.

[1323] "Example 2: Combining Emotion Engines"

[1324] (Claim 1)

[1325] a means for constructing a database for storing data relating to sales techniques;

[1326] A means for recording and recording the user's sales performance;

[1327] A means for transmitting video and audio data to a server;

[1328] The server analyzes the video and audio data to evaluate sales techniques, facial expressions, and tone of voice.

[1329] means for generating feedback based on the analysis results;

[1330] means for transmitting feedback to a user terminal;

[1331] a means for using an emotion engine to analyze the user's emotions;

[1332] A means for reflecting the sentiment analysis results in the generated feedback;

[1333] A system including:

[1334] (Claim 2)

[1335] 2. The system according to claim 1, further comprising means for allowing a user to view the feedback and retry the sales performance in consideration of the sentiment analysis results.

[1336] (Claim 3)

[1337] The system described in claim 1, characterized in that a deep learning model is used to analyze video and audio data.

[1338] "Application example 2 when combining emotion engines"

[1339] (Claim 1)

[1340] a means for constructing a database for storing data relating to sales techniques;

[1341] A means for recording and recording the user's sales performance;

[1342] A means for transmitting video and audio data to a server;

[1343] The server analyzes the video and audio data to evaluate sales techniques, facial expressions, and tone of voice.

[1344] means for generating feedback based on the analysis results;

[1345] means for transmitting feedback to a user terminal;

[1346] The system will support learning customer service skills in stores and customer service work,

[1347] A means for recognizing the user's emotions and reflecting them in feedback;

[1348] A system including:

[1349] (Claim 2)

[1350] 10. The system of claim 1, further comprising means for a user to view the feedback and attempt to re-engage the customer based on improvements.

[1351] (Claim 3)

[1352] The system described in claim 1 is characterized in that it uses a deep learning model to analyze video and audio data, and generates specific feedback based on the analysis results using an AI model. [Explanation of symbols]

[1353] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for constructing a database for storing data relating to sales techniques; A means for recording and recording the user's sales performance; A means for transmitting video and audio data to a server; The server analyzes the recorded data and evaluates the sales techniques, facial expressions, and tone of voice. means for generating feedback based on the analysis results; means for transmitting feedback to a user terminal; A system including:

2. 10. The system of claim 1, further comprising means for a user to view the feedback and reattempt the sales performance based on improvements.

3. The system of claim 1, characterized in that a deep learning model is used to analyze video and audio data.

Images