Data processing device and data processing system

Abstract

This involves simultaneously handling customer service and managing the stress levels of sales representatives. [Solution] The data processing device includes an emotion recognition unit that recognizes the emotional state of the customer based on voice data including the customer's spoken voice and image data including the customer's facial expressions; a stress level derivation unit that derives the stress level of the sales representative based on the sales representative's biometric data; and a feedback generation unit that generates effective feedback for promoting sales activities with the customer based on the recognized emotional state of the customer and the derived stress level of the sales representative.

Description

Technical Field

[0001] The technology of the present disclosure relates to a data processing apparatus and a data processing system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there is room for improvement in collecting the customer's biometric data.

Means for Solving the Problems

[0005] [ The data processing apparatus according to the disclosed technology includes an emotion recognition unit that recognizes the emotional state of a customer based on voice data including the customer's spoken voice and image data including the customer's facial expression, a stress level derivation unit that derives the stress level of a salesperson based on the biometric data of the salesperson, and a feedback generation unit that generates effective feedback for promoting the sales activities for the customer well using a data generation model based on the recognized emotional state of the customer and the derived stress level of the salesperson.

[0006] The feedback generation unit may use the data generation model to generate advice on how to proceed with the business negotiation as feedback, based on the recognized emotional state of the customer.

[0007] If the derived stress level of the sales representative is above a threshold, the feedback generation unit may use the data generation model to generate a method for reducing the stress level as feedback.

[0008] The feedback generation unit may, if the correlation between the recognized emotional state of the customer and the derived stress level of the sales representative is above a certain level, use the data generation model to generate a warning to the sales representative as feedback.

[0009] The data processing device may further include a report generation unit that uses the data generation model to generate a report that includes areas for improvement and recommended approaches for the next business negotiation, based on a negotiation record that includes the content of the conversation between the customer and the sales representative during the negotiation, the customer's emotional state during the negotiation, the sales representative's stress level during the negotiation, and the content of the feedback.

[0010] The data processing system relating to the disclosed technology includes a wearable terminal device worn by the sales representative. The terminal device includes a macrophone that captures the customer's spoken voice and generates the voice data, a camera that photographs the customer and generates the image data, and a sensor that acquires the biometric data.

[0011] The data processing device may include a communication unit that transmits audio data including the content of the feedback to the terminal device. The terminal device may include a speaker that outputs audio data including the content of the feedback. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system. [Figure 2] This is a conceptual diagram showing an example of the main functions of a data processing device and a necklace-type terminal. [Figure 3] This is a side view showing the configuration of a necklace-type terminal. [Figure 4] This is a top view showing the configuration of a necklace-type terminal. [Figure 5] The functional configuration of the control unit of the necklace-type terminal is shown in general terms. [Figure 6] The functional configuration of a specific processing unit of a data processing device is shown in general terms. [Figure 7] This diagram outlines an example of the operation flow of a specific process performed by a data processing device. [Figure 8] This diagram schematically illustrates the flow of various types of data transmitted and received between a necklace-type terminal and a data processing device. [Figure 9] The functional configuration of a specific processing unit of a data processing device is shown in general terms. [Figure 10] This diagram outlines an example of the operation flow of a specific process performed by a data processing device. [Modes for carrying out the invention]

[0013] [First Embodiment] Hereinafter, an example of an embodiment of the data processing device, data processing method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, let's explain the terminology used in the following explanation.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit), etc.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory where information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface that includes a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a necklace-type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of this disclosure, and the necklace-type terminal 14 is an example of a "necklace-type terminal" according to the technology of this disclosure.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The necklace-type terminal 14 includes a computer 36, a microphone 38, a sensor 39, a speaker 40, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 38, speaker 40, and camera 42 are also connected to the bus 52.

[0024] The user 20 wearing the necklace-type terminal 14 may be, for example, a patient whose health condition is being diagnosed, or a regular user.

[0025] The microphone 38 picks up the voice emitted by the user 20, who is wearing the necklace-type terminal 14, as well as sounds around the user 20. The microphone 38 also receives instructions from the user 20 by receiving the voice emitted by the user 20. The microphone 38 captures the voice emitted by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs audio according to the instructions from the processor 46. The speaker 40 is, for example, a directional speaker and outputs audio towards the user 20's ears.

[0026] Sensor 39 is a sensor that detects biometric data of the user 20, who is wearing the necklace-type terminal. For example, sensor 39 may be a heart rate sensor or a blood oxygen sensor.

[0027] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the necklace-type terminal 14.

[0030] As shown in Figure 2, in the data processing device 12, specific processing is performed by the processor 28. The storage 32 stores a specific processing program 56. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58. The data generation model 58 is used by the specific processing unit 290. The storage 32 also includes a data storage unit 59.

[0032] In the necklace-type terminal 14, data acquisition processing is performed by the processor 46. The storage 50 stores the data acquisition program 60. The processor 46 reads the data acquisition program 60 from the storage 50 and executes the read data acquisition program 60 on the RAM 48. The data acquisition processing is realized by the processor 46 operating as a control unit 46A according to the data acquisition program 60 executed on the RAM 48.

[0033] As shown in Figures 3 and 4, the necklace-type terminal 14 includes multiple microphones 38, multiple sensors 39, multiple speakers 40, and multiple cameras 42. Figures 3 and 4 show an example where two microphones 38 are positioned in front of the user 20 when the user 20 wears the necklace-type terminal 14. They also show an example where two sensors 39 are positioned to the right and left of the user 20 when the user 20 wears the necklace-type terminal 14. They also show an example where two speakers 40 are positioned to the right rear and left rear of the user 20 when the user 20 wears the necklace-type terminal 14. They also show an example where two cameras 42 are positioned to the right front and left front of the user 20 when the user 20 wears the necklace-type terminal 14. Finally, they show an example where two sensors 39 are positioned inside the necklace-type terminal 14 so as to contact the user 20's neck when the user 20 wears the necklace-type terminal 14.

[0034] Next, we will explain the processing of the control unit 46A when the necklace-type terminal 14 performs data collection processing to collect data.

[0035] In this embodiment, the data collection process collects the user's biometric data in real time. Furthermore, it collects not only biometric data but also all surrounding environment data. This makes it possible to detect early signs of conditions such as Alzheimer's disease and dementia. It also allows for monitoring of the user's health status (e.g., heart disease).

[0036] As shown in Figure 5, the control unit 46A includes a data acquisition unit 100 and a communication unit 102.

[0037] The data acquisition unit 100 collects the outputs of the microphone 38, sensor 39, and camera 42, respectively.

[0038] The communication unit 102 transmits the outputs of the microphone 38, sensor 39, and camera 42, which are collected by the data acquisition unit 100, to the data processing unit 12.

[0039] Next, we will describe the processing of the specific processing unit 290 when the data processing device 12 performs specific processing to obtain a response corresponding to a user utterance.

[0040] In the specific processing in this embodiment, a response corresponding to the user utterance picked up by the microphone 38 of the necklace-type terminal 14 is obtained using the data generation model 58.

[0041] As shown in Figure 6, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0042] The input unit 292 stores the outputs of the microphone 38, sensor 39, and camera 42, respectively, received from the necklace-type terminal 14, in the data storage unit 59.

[0043] The input unit 292 acquires user utterances received by the necklace-type terminal 14. Specifically, it acquires user utterances picked up by the microphone 38 of the necklace-type terminal 14.

[0044] The processing unit 294 performs specific processing using the data generation model 58. Specifically, it inputs a prompt including user utterances to the data generation model 58 and obtains a generation result. At this time, the outputs of the sensor 39 and camera 42 collected by the data acquisition unit 100 may also be included in the prompt.

[0045] The output unit 296 transmits the result of the specific processing to the necklace-type terminal 14. In the necklace-type terminal 14, the control unit 46A causes the speaker 40 to output the result of the specific processing. In this way, a response corresponding to the user utterance picked up by the microphone 38 is output to the user 20 by the speaker 40. The microphone 38 further acquires the user utterance in response to the result of the specific processing. The control unit 46A transmits the audio data indicating the user utterance acquired by the microphone 38 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the user utterance.

[0046] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0047] The outputs of the microphone 38, sensor 39, and camera 42 stored in the data storage unit 59 are used, for example, to diagnose the health status of user 20. In this case, the outputs of the microphone 38, sensor 39, and camera 42 stored in the data storage unit 59 may be transmitted to a terminal on the medical institution side. Alternatively, the data processing device 12 may analyze the outputs of the microphone 38, sensor 39, and camera 42 stored in the data storage unit 59 to diagnose the health status of user 20.

[0048] Next, the operation of the data processing system 10 will be explained.

[0049] First, let's explain an example of the data collection process flow.

[0050] When user 20 is wearing the necklace-type terminal 14, the data acquisition unit 100 sequentially collects the outputs of the microphone 38, sensor 39, and camera 42. The communication unit 102 sequentially transmits the outputs of the microphone 38, sensor 39, and camera 42 collected by the data acquisition unit 100 to the data processing unit 12.

[0051] Next, an example of the flow of a specific process will be explained with reference to Figure 7. Here, the input unit 292 of the data processing device 12 sequentially acquires the outputs of the microphone 38, sensor 39, and camera 42 received from the necklace-type terminal 14 and stores them in the data storage unit 59.

[0052] In step S300, the processing unit 294 determines whether a predetermined trigger condition is met. Specifically, the trigger condition may be that the user utterance picked up by the microphone 38 contains a specific word (for example, the name of the agent installed in the necklace-type terminal 14) or a phrase (for example, "Hi! XX" (where XX is the name of the agent)).

[0053] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 terminates the specific processing.

[0054] In step S301, the processing unit 294 generates a prompt by adding an instruction sentence to the text representing the user utterance picked up by the microphone 38, in order to obtain the result of a specific process.

[0055] For example, a prompt such as "The user is speaking as follows: XXX. Please respond as the agent." (XXX is the user's speech) can be generated. Alternatively, the outputs of sensor 39 and camera 42 can be added to the prompt to generate a prompt such as "This is biometric data representing the user's heart rate and video data representing the user's surroundings. The user is also speaking as follows: XXX. Please respond as the agent." (XXX is the user's speech)

[0056] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58 and obtains the result of a specific process based on the output of the data generation model 58.

[0057] In step S304, the output unit 296 outputs the result of the specific processing to the necklace-type terminal 14 and terminates the specific processing.

[0058] [Second Embodiment] In sales roles, managing the stress levels of sales representatives is just as important as customer service. Prolonged high stress levels can lead to decreased performance and negatively impact the success rate of business negotiations. Therefore, it is desirable to manage both customer service and sales representative stress simultaneously. The data processing system according to the second embodiment of the disclosed technology has a function to manage sales representative stress while supporting sales activities using a necklace-type terminal. Details of the data processing system according to the second embodiment are described below.

[0059] Figure 8 is a schematic diagram showing the flow of various data transmitted and received between the necklace-type terminal 14 and the data processing device 12. In this embodiment, the necklace-type terminal 14 is worn by a sales representative 401 who conducts sales activities with a customer 400. The necklace-type terminal 14 is an example of a "wearable terminal device" in the disclosed technology.

[0060] The camera 42 on the necklace-type terminal 14 photographs the customer 400 and generates image data including the customer 400's facial expression. The microphone 38 on the necklace-type terminal 14 captures the customer 400's speech and generates audio data. The sensor 39 on the necklace-type terminal 14 generates biometric data including at least one of the following: heart rate, blood pressure, skin temperature, and skin potential of the sales representative 401.

[0061] Image data, voice data, and biometric data are collected by the data collection unit 100 of the necklace-type terminal 14. The collected image data, voice data, and biometric data are transmitted to the data processing device 12 by the communication unit 102 of the necklace-type terminal 14.

[0062] The processor 28 of the data processing device 12 operates as a specific processing unit 290A by executing a specific processing program 56 on the RAM 30. As shown in Figure 9, the specific processing unit 290A includes an acquisition unit 501, an emotion recognition unit 502, a stress level derivation unit 503, a feedback generation unit 504, a report generation unit 505, and a communication unit 506.

[0063] The acquisition unit 501 acquires image data, voice data, and biometric data transmitted from the necklace-type terminal 14, and stores this data in the data storage unit 59.

[0064] The emotion recognition unit 502 recognizes the emotional state of customer 400 based on audio data including the customer's speech and image data including the customer's facial expressions. The emotion recognition unit 502 recognizes the emotional state of customer 400 using, for example, an acoustic analysis model and an image analysis model learned by machine learning.

[0065] The acoustic analysis model analyzes audio data, including the speech of 400 customers, to determine the emotional state of each customer by analyzing the volume, intonation, and other characteristics of their voices, capturing changes in voice features and frequencies. The image analysis model analyzes image data, including the facial expressions of 400 customers, to determine their emotional state by analyzing facial muscle movements, eye movements, pupil dilation, and other characteristics.

[0066] The emotion recognition unit 502 outputs the recognition result of the customer 400's emotional state by integrating the analysis results from the acoustic analysis model and the image analysis model. The emotion recognition unit 502 may also output data that quantifies the intensity of each emotional element, including, for example, "interest," "dissatisfaction," "surprise," and "anger," as the recognition result of the customer 400's emotional state.

[0067] The stress level derivation unit 503 derives the stress level of the sales representative 401 based on the biometric data acquired for the sales representative 401. The stress level is reflected in heart rate, blood pressure, skin temperature, and skin potential. The stress level derivation unit 503 outputs numerical data of the sales representative 401's stress level based on biometric data including at least one of the following: heart rate, blood pressure, skin temperature, and skin potential.

[0068] The feedback generation unit 504 generates effective feedback to facilitate sales activities for customer 400, based on the emotional state of customer 400 recognized by the emotion recognition unit 502 and the stress level of sales representative 401 derived by the stress level derivation unit 503. The feedback generation unit 504 generates feedback by inputting a prompt into the data generation model 58 that includes the recognized emotional state of customer 400 and the derived stress level of sales representative 401, instructing it to generate effective feedback to facilitate sales activities for customer 400. For example, the prompt "The customer's emotional state is 'dissatisfied'. The sales representative's stress level is '8'. To facilitate sales activities for this customer, please generate effective feedback for the sales representative" is input into the data generation model 58. At this time, for example, feedback such as "Please seek support from other sales representatives" is generated in real time.

[0069] The feedback generation unit 504 generates advice on how to proceed with the business negotiation as feedback by inputting a prompt to the data generation model 58 that instructs it to generate advice on how to proceed with the business negotiation based on the recognition result of the customer 400's emotional state. For example, if the recognized emotional state of the customer 400 is "dissatisfied", the feedback generation unit 504 inputs the prompt "The customer's emotional state is 'dissatisfied'. Please generate advice on how to proceed with the business negotiation" to the data generation model 58. At this time, the data generation model 58 generates audio data as feedback that includes content such as "The customer is feeling dissatisfied. Let's change the topic."

[0070] Furthermore, if the derived stress level of the sales representative 401 is above a threshold, the feedback generation unit 504 inputs a prompt to the data generation model 58 instructing it to generate countermeasures to reduce the stress level, thereby generating countermeasures to reduce the stress level as feedback. For example, the prompt "The sales representative's stress level is '8'. Please generate countermeasures to reduce the stress level" is input to the data generation model 58. At this time, the feedback generation unit 504 generates audio data including a breathing guide or relaxation audio as feedback.

[0071] The feedback generation unit 504 evaluates the correlation between the recognized emotional state of the customer 400 and the derived stress level of the sales representative 401. For example, if the stress level of the sales representative 401 rises in line with the increase in the customer 400's negative emotions, the correlation between the customer 400's emotional state and the sales representative 401's stress level is evaluated as high. If the correlation between the customer 400's emotional state and the sales representative 401's stress level is above a certain level, the feedback generation unit 504 inputs a prompt to the data generation model 58 instructing it to generate a warning for the sales representative, thereby generating a warning for the sales representative as feedback. For example, if the correlation is above a certain level, the feedback generation unit 504 inputs the prompt, "The sales representative's stress level is rising in line with the customer's increasing negative emotions. Please generate a warning for this sales representative," to the data generation model 58. At this point, the data generation model 58 generates audio data as feedback that includes content such as, "The customer's emotions and your stress level are influencing each other. By relaxing, you can positively change the customer's emotions."

[0072] The sales negotiation record, which includes the content of the conversation between customer 400 and sales representative 401 during the negotiation, customer 400's emotional state, sales representative 401's stress level, and the content of the feedback generated by the feedback generation unit 504, is stored in the storage 32 of the data processing device 12. After the negotiation is completed, the report generation unit 505 generates a report including areas for improvement and recommended approaches for the next negotiation by inputting a prompt to the data generation model 58, which instructs the model 58 to generate a report including areas for improvement and recommended approaches for the next negotiation based on the sales negotiation record stored in the storage 32. The report generation unit 505 generates the report by inputting the sales negotiation record and the prompt to the data generation model 58. The report generation unit 505 may also generate the report using customer information, including past negotiation history, customer preferences, and past purchase history, which can be obtained from the database stored in the storage 32.

[0073] The communication unit 506 transmits to the necklace-type terminal 14 audio data containing the feedback content generated by the feedback generation unit 504 and audio data containing the report content generated by the report generation unit 505.

[0074] Figure 10 is a flowchart showing an example of the flow of a specific processing operation performed by the specific processing unit 290A of the data processing device 12.

[0075] In step S310, the acquisition unit 501 acquires image data including the facial expression of the customer 400, voice data including the customer 400's spoken voice, and biometric data of the sales representative 401 transmitted from the necklace-type terminal 14, and stores this data in the data storage unit 59.

[0076] In step S311, the emotion recognition unit 502 recognizes the emotional state of the customer 400 based on the audio data and image data acquired in step S310.

[0077] In step S312, the stress level derivation unit 503 derives the stress level of the sales representative 401 based on the biological data acquired in step S310.

[0078] In step S313, the feedback generation unit 504 uses the data generation model 58 to generate effective feedback for promoting sales activities to the customer 400, based on the emotional state of the customer 400 recognized in step S311 and the stress level of the sales representative 401 derived in step S312.

[0079] In step S314, the communication unit 506 transmits audio data containing the feedback generated in step S313 to the necklace-type terminal 14. The speaker 40 of the necklace-type terminal 14 outputs audio containing the feedback.

[0080] In step S315, the report generation unit 505 determines whether the business negotiation has ended. The report generation unit 505 may determine whether the business negotiation has ended based, for example, on the conversation between the customer 400 and the sales representative 401 transmitted through the microphone of the necklace-type terminal 14. If it is determined that the business negotiation has not ended, the process returns to step S310. If it is determined that the business negotiation has ended, the process proceeds to step S316. The business negotiation record, which includes the content of the conversation between the customer 400 and the sales representative 401 during the business negotiation, the emotional state of the customer 400 during the business negotiation, the stress level of the sales representative 401 during the business negotiation, and the content of the feedback generated by the feedback generation unit 504, is stored in the storage 32 of the data processing device 12.

[0081] In step S316, the report generation unit 505 generates a report based on the opportunity records stored in the storage 32, using the data generation model 58, which includes areas for improvement and recommended approaches for the next opportunity. The report generation unit 505 may also generate the report using customer information, including past opportunity history, customer preferences, and past purchase history, which can be obtained from the database stored in the storage 32.

[0082] In step S317, the communication unit 506 transmits audio data containing the contents of the report generated in step S316 to the necklace-type terminal 14. The speaker 40 of the necklace-type terminal 14 outputs audio containing the contents of the report.

[0083] As described above, the data processing device 12 according to this embodiment includes an emotion recognition unit 502 that recognizes the emotional state of a customer 400 based on voice data including the customer 400's spoken voice and image data including the customer 400's facial expressions; a stress level derivation unit 503 that derives the stress level of a sales representative 401 based on the sales representative's biometric data; and a feedback generation unit 504 that uses a data generation model 58 to generate effective feedback for promoting sales activities with the customer based on the recognized emotional state of the customer 400 and the derived stress level of the sales representative.

[0084] The feedback generation unit 504 uses the data generation model 58 to generate feedback on how to proceed with the sales negotiation, based on the emotional state of the recognized customer 400. For example, if the emotional state of the recognized customer 400 is "dissatisfied," the sales representative 401 is provided with an audio message containing content such as, "The customer is dissatisfied. Let's change the topic." This allows the sales representative to be informed in real time of the optimal approach tailored to the progress of the sales negotiation.

[0085] The feedback generation unit 504, if the derived stress level of the sales representative is above a threshold, uses the data generation model 58 to generate feedback on coping strategies to reduce the stress level. For example, if sales representative 401 has a high stress level, an intervention such as a breathing exercise guide or relaxation audio is implemented. This reduces the stress of sales representative 401 and contributes to the smooth progress of the business negotiation.

[0086] The feedback generation unit 504 generates a warning as feedback for the sales representative 401 using the data generation model 58 if the correlation between the recognized emotional state of the customer 400 and the derived stress level of the sales representative 401 is above a certain level. For example, if the correlation is above a certain level, the sales representative is provided with an audio message containing content such as, "The customer's emotions and your stress level are influencing each other. By relaxing, you can change the customer's emotions in a positive direction." This makes it possible to reduce the stress of the sales representative 401 while guiding the emotional state of the customer 400 in a positive direction.

[0087] According to the data processing system 10 of this embodiment, it becomes possible to handle customer inquiries and manage the stress levels of sales representatives simultaneously.

[0088] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0089] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes multiple computers, including computer 22.

[0090] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0091] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0092] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0093] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0094] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0095] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0096] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0097] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0098] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference. [Explanation of Symbols]

[0099] 10 Data Processing Systems 12 Data Processing Devices 14 Necklace-type terminal 38 Microphones 39 Sensors 40 speakers 42 cameras 46A Control Unit 100 Data Acquisition Unit 102 Communications Department 290 Specific Processing Unit 292 Input section 294 Processing Unit 296 Output section< / url:>

Claims

1. An emotion recognition unit recognizes the customer's emotional state based on audio data including the customer's spoken voice and image data including the customer's facial expressions. A stress level derivation unit that derives the stress level of a sales representative based on the sales representative's biometric data, A feedback generation unit generates effective feedback for promoting sales activities to the customer, using a data generation model based on the recognized emotional state of the customer and the derived stress level of the sales representative. A data processing device that includes a data processing device.

2. The feedback generation unit generates advice on how to proceed with the business negotiation as feedback, using the data generation model based on the recognized emotional state of the customer. The data processing device according to claim 1.

3. If the derived stress level of the sales representative is above a threshold, the feedback generation unit uses the data generation model to generate the feedback, which is a method of action to reduce the stress level. The data processing device according to claim 1.

4. The feedback generation unit, when it determines that the correlation between the recognized customer's emotional state and the derived sales representative's stress level is above a certain level, uses the data generation model to generate a warning to the sales representative as feedback. The data processing device according to claim 1.

5. The system further includes a report generation unit that uses the data generation model to generate a report that includes areas for improvement and recommended approaches for the next business negotiation, based on a negotiation record that includes the content of the conversation between the customer and the sales representative during the negotiation, the customer's emotional state during the negotiation, the sales representative's stress level during the negotiation, and the content of the feedback. The data processing device according to claim 1.

6. A data processing device according to any one of claims 1 to 5, A data processing system including a wearable terminal device to be worn by the aforementioned sales representative, The aforementioned terminal device is A macrophone that captures the customer's spoken voice and generates the voice data, A camera that photographs the customer and generates the image data, A sensor that acquires the aforementioned biological data, A data processing system that includes this.

7. The data processing device includes a communication unit that transmits audio data including the content of the feedback to the terminal device. The terminal device includes a speaker that outputs audio data including the content of the feedback. The data processing system according to claim 6.

Images