System

Abstract

A system is provided.SOLUTION: A system comprising: means for transcribing audio data of a meeting in real-time; means for analyzing the transcribed data to assess a sentiment or reaction of a customer; means for generating an appropriate transcript based on the sentiment or reaction of the customer; and means for presenting the generated transcript to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern business, the ability to respond quickly and appropriately during conversations and meetings with customers is extremely important. However, it is difficult for salespeople to accurately read customers' reactions and emotions and deliver optimal conversations in real time. Furthermore, when customers are not convinced, they often become at a loss for words and fall silent. This can slow down the progress of sales negotiations and risk missing business opportunities. The present invention solves these problems and provides a system that supports salespeople in efficiently conducting conversations. [Means for solving the problem]

[0005] The present invention is a system that includes a means for transcribing conference audio data in real time, a means for analyzing the transcript data and evaluating customer emotions and reactions, a means for generating an appropriate talk script based on the customer emotions and reactions, and a means for presenting the generated talk script to a user. This system enables sales representatives to quickly respond appropriately to customer reactions, thereby smoothing the progress of business negotiations. Furthermore, by providing a means for continuously analyzing the progress of a conference and dynamically updating the talk script based on the customer's enthusiasm, the system can support more flexible and effective sales activities. Furthermore, by analyzing the transcript data using natural language processing technology, it is possible to accurately grasp customer emotions and reactions.

[0006] "Conference audio data" refers to all audio information generated during a conference, and in particular data including the content of speech.

[0007] "Real-time" refers to data processing and manipulation occurring in real time, with little or no delay.

[0008] "Transcription" is the process of converting audio data into text form.

[0009] "Analysis" refers to the detailed examination of collected data to extract specific patterns and meanings.

[0010] "Customer emotions and reactions" refers to the psychological state and reactions that can be read from the customer's comments and attitude during the meeting.

[0011] "Evaluation" is the act of making a judgment based on specific criteria, based on analyzed data.

[0012] A "talk script" is a scenario that shows appropriate words and conversation flow for a sales representative to use when speaking to a customer.

[0013] "Generation" is the process of creating new data or content based on necessary information.

[0014] "Presenting" refers to the act of displaying or communicating the generated information to a user.

[0015] "User" refers to anyone who operates or uses this system, and primarily includes sales personnel.

[0016] "Sequentially" refers to performing processing sequentially at regular intervals.

[0017] "Dynamic update" refers to changing or modifying information in real time or as needed in response to changing circumstances.

[0018] "Natural language processing technology" is a technical field that enables computers to understand and analyze human language. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system for enabling a sales representative to respond quickly and appropriately in meetings and business negotiations with customers. The system includes the following components.

[0041] System Configuration

[0042] 1. Terminal

[0043] The device records the audio of the meeting in real time.

[0044] The device calls a speech recognition API to transcribe the recorded audio in real time.

[0045] The device sequentially buffers the generated transcription data and sends it to the server.

[0046] 2. Server

[0047] The server receives the transcription data sent from the terminal.

[0048] The server uses natural language processing technology to analyze the transcription data.

[0049] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[0050] The server sends the generated talk script to the terminal.

[0051] 3. User (Salesperson)

[0052] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[0053] Overview of program processing

[0054] 1. Acquiring and transcribing audio data

[0055] The device collects audio data during the meeting and transcribes it in real time using a speech recognition API.

[0056] The device buffers the transcription data sequentially and transmits it to the server at regular time-sliding intervals.

[0057] 2. Analysis of the transcription data

[0058] The server receives the transcription data sent from the terminal.

[0059] The server applies natural language processing techniques to the received data and analyzes the text.

[0060] Example: Analyze the statement "I think this product is too expensive" to recognize negative evaluations of price.

[0061] 3. Evaluating emotions and reactions

[0062] The server evaluates the customer's emotions and reactions based on the analysis results.

[0063] Example: The statement "it's too expensive" is judged to indicate a negative sentiment.

[0064] 4. Generate a talk script

[0065] The server generates an appropriate talk script based on the evaluation result.

[0066] Example: Generates a script that says, "Price is certainly an important factor, but our products offer excellent long-term value for money."

[0067] 5. Send and present the talk script

[0068] The server sends the generated talk script to the terminal.

[0069] The terminal presents the talk script to the user visually or audibly.

[0070] Specific examples

[0071] Initial Setup and Starting a Meeting

[0072] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product."

[0073] The device will turn on the voice recognition function and begin recording the meeting audio.

[0074] Real-time transcription and analysis

[0075] A customer says, "I think the price of this product is too high."

[0076] The device transcribes this speech in real time and sends the text data to a server.

[0077] The server analyzes the text data and evaluates the customer's sentiment as negative.

[0078] Generate and present talk scripts

[0079] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[0080] The server sends the generated script to the terminal, which displays it to the user.

[0081] User response

[0082] Using the script provided, the user responds, "Price is certainly an important point, but our products offer excellent long-term cost performance," and continues the conversation with the customer.

[0083] In this way, the present invention provides an effective tool to assist salespeople in responding quickly and appropriately during a meeting.

[0084] The processing flow will be explained below.

[0085] Step 1:

[0086] The user starts the system and sets the goal and main agenda of the meeting. For example, the user enters "concluding a contract for a new product" as the goal.

[0087] Step 2:

[0088] The device will turn on the voice recognition function and begin recording the meeting audio in real time.

[0089] Step 3:

[0090] The device sends the recorded voice data to the speech recognition API and converts it into text data in real time. The converted text data is buffered sequentially.

[0091] Step 4:

[0092] The device sends buffered transcription data to the server at regular time-sliding intervals.

[0093] Step 5:

[0094] The server receives the transcription data sent from the device and places the received data in a queue for analysis.

[0095] Step 6:

[0096] The server uses natural language processing technology to analyze the transcription data, for example, analyzing a customer statement in the text, "I think the price of this product is too high," to recognize a negative reaction to the price.

[0097] Step 7:

[0098] The server evaluates the customer's emotions and reactions based on the analysis results. Comments such as "too expensive" are evaluated negatively, and the server determines that the enthusiasm for the meeting is declining.

[0099] Step 8:

[0100] Based on the evaluation results, the server generates an appropriate talk script, such as, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[0101] Step 9:

[0102] The server sends the generated talk script to the terminal, which is then used to continue the conference.

[0103] Step 10:

[0104] The terminal displays the generated talk script to the user, who then uses this script to speak to the customer.

[0105] Step 11:

[0106] The user responds, "Price is certainly an important factor, but our products offer excellent long-term value for money," and continues the conversation.

[0107] Step 12:

[0108] The device records the speech again and repeats the process from step 3. Each time new transcription data is generated, it is sent sequentially to the server, where the process of analysis, evaluation, and script generation continues.

[0109] In this way, the system helps manage meetings in real time, enabling salespeople to effectively serve customers.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] The lack of effective support tools that enable sales representatives to respond quickly and appropriately during meetings and negotiations with customers is an issue. Specifically, the lack of real-time feedback on customer comments and the provision of appropriate conversation scripts can reduce the success rate of sales negotiations.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes means for acquiring conference audio data in real time using a voice input device, means for transcribing in real time using a voice recognition API, means for sequentially buffering the transcribed data and transmitting it to the server at regular time slide intervals, means for receiving the transcribed data and analyzing it using natural language processing technology, means for evaluating the customer's emotions and reactions based on the analysis results, means for generating an appropriate talk script using a generative AI model based on the evaluation results, and means for presenting the generated talk script to the user. This enables sales representatives to respond quickly and appropriately to customer comments and emotions during meetings and lead sales negotiations to success.

[0115] The "voice input device" is hardware for acquiring conference voice data in real time.

[0116] A "voice recognition API" is a software interface for converting acquired voice data into text data.

[0117] "Buffering" is a process of temporarily storing transcription data and sending it to a server at regular time-slide intervals.

[0118] "Natural language processing technology" refers to algorithms and technologies for analyzing text data and understanding meaning and emotions.

[0119] A "generative AI model" is an artificial intelligence algorithm that generates an appropriate talk script based on the analysis results.

[0120] A "talk script" is a conversation script that sales representatives can use as a reference, generated based on the customer's emotions and reactions.

[0121] "Sequential" means performing a series of data or processes one step at a time in order.

[0122] The present invention provides a system that enables salespeople to respond quickly and appropriately in meetings and business negotiations with customers. This system operates in cooperation with terminals, a server, and users.

[0123] System Configuration

[0124] 1. Terminal

[0125] The terminal uses a voice input device to capture the audio data of the meeting in real time. The hardware used here is a regular smartphone or tablet.

[0126] The device uses speech recognition APIs such as Google® Cloud Speech-to-Text API and Microsoft® Azure® Cognitive Services to perform real-time transcription.

[0127] The terminal has a communication function for sequentially buffering the transcription data and transmitting it to the server at regular time-sliding intervals.

[0128] Examples:

[0129] When a user starts a meeting, they launch the application on their device and turn on the speech recognition function. Each utterance made during the meeting is transcribed in real time and continuously sent to the server.

[0130] Example prompt sentence:

[0131] "Transcribe meeting audio in real time."

[0132] 2. Server

[0133] The server receives the transcription data sent from the device. A regular web server is used here, and an RDBMS such as MySQL (registered trademark) or PostgreSQL is used as the database.

[0134] The server analyzes the received text data using a Python natural language processing library (e.g., NLTK or SpaCy).

[0135] The server evaluates the customer's emotions and reactions based on the analysis results. For example, it determines that a comment such as "too expensive" indicates a negative evaluation of the price.

[0136] The server uses a generative AI model (e.g., GPT-3 (registered trademark)) to generate an appropriate talk script.

[0137] Examples:

[0138] If a customer says during a meeting, "I think the price of this product is too high," the server analyzes the comment in real time, detects the negative sentiment of "high price," and then generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance."

[0139] Example prompt sentence:

[0140] "Generate a talking script to use when a customer has a negative comment about the price."

[0141] 3. User (Salesperson)

[0142] The user can proceed with the conversation by referring to the talk script provided by the device, which can either display the script visually or read it aloud.

[0143] Examples:

[0144] During the meeting, the user responds to the customer using the script displayed on the terminal, which reads, "Price is certainly an important point. However, our products offer excellent long-term cost performance."

[0145] Example prompt sentence:

[0146] "Show the generated talk script to the user."

[0147] With the above configuration, the system of the present invention enables salespeople to respond quickly and appropriately to customer comments and feelings during a meeting, leading to successful business negotiations.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Step 1:

[0150] The terminal acquires the audio data of the conference using an audio input device. The input is the audio data of the conference, and the output is the acquired audio data. Specifically, the user launches an application on the terminal and turns on the voice recognition function. The terminal records the conference audio through the microphone.

[0151] Step 2:

[0152] The device transcribes the acquired audio data in real time using a speech recognition API. The input is the audio data from step 1, and the output is the transcribed text data. This operation uses the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services. Specifically, the device sends the audio data to the API and receives the text data returned by the API.

[0153] Step 3:

[0154] The device buffers the transcription data sequentially and sends it to the server at regular time-sliding intervals. The input is the transcription data from step 2, and the output is the text data sent to the server. Specifically, the device buffers the text data obtained every 5 seconds and sends it to the server via an HTTP POST request.

[0155] Step 4:

[0156] The server receives the transcription data sent from the device. The input is the text data sent in step 3, and the output is the received transcription data. Specifically, the server receives the HTTP request and writes the data to a database (MySQL or PostgreSQL).

[0157] Step 5:

[0158] The server analyzes the received transcription data using natural language processing technology. The input is the data received in step 4, and the output is the analysis results. This operation uses Python's NLTK and SpaCy libraries. Specifically, the server tokenizes the text data and performs sentiment analysis.

[0159] Step 6:

[0160] The server evaluates the customer's emotions and reactions based on the analysis results. The input is the analysis result from step 5, and the output is the evaluation result. Specifically, the server checks keywords such as "high" and "negative" against a dictionary and sets a negative flag.

[0161] Step 7:

[0162] Based on the evaluation results, the server uses a generative AI model to generate an appropriate talk script. The input is the evaluation result from step 6, and the output is the generated talk script. This operation uses a generative AI model such as OpenAI's (registered trademark) GPT-3. Specifically, the server inputs a prompt statement (e.g., "Please generate a talk script that can be used when making negative comments about prices") into the AI ​​model and receives the script generated by the AI ​​model.

[0163] Step 8:

[0164] The server sends the generated talk script to the terminal. The input is the talk script from step 7, and the output is the script sent to the terminal. Specifically, the server sends the talk script to the terminal using an HTTP POST request.

[0165] Step 9:

[0166] The terminal presents the received talk script to the user. The input is the talk script from step 8, and the output is the script presented to the user. Specifically, the terminal displays the script in the application's display area or provides a function to read it aloud.

[0167] (Application example 1)

[0168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0169] In traditional sales negotiations and customer service, it was difficult for salespeople and store clerks to accurately grasp what customers were saying and how they were feeling, and then respond appropriately immediately. Furthermore, because conversations were recorded and analyzed manually, real-time support was difficult, leading to inconsistencies in the quality of customer service. This resulted in issues such as lower customer satisfaction and a lower success rate for sales negotiations.

[0170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0171] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for collecting audio data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time, thereby enabling salespeople or store clerks to receive appropriate feedback and responses in real time during conversations with customers.

[0172] "Means for transcribing meeting audio data in real time" is a technology that converts audio data collected during meetings or business negotiations into text data in real time.

[0173] "Means of analyzing transcribed data and evaluating customer emotions and reactions" refers to a technology that analyzes transcribed data using natural language processing technology and evaluates customer emotions and reactions.

[0174] "Means for generating appropriate talk scripts based on customers' emotions and reactions" refers to technology that automatically generates appropriate response content based on customers' emotions and reactions obtained through analysis.

[0175] The "means for presenting the generated talk script to the user" is a technique for immediately providing the generated response content to the user visually or audibly.

[0176] "Means for collecting voice data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time" refers to technology that collects voice data in real time during business negotiations or customer interactions, analyzes the data, and instantly generates feedback and responses based on the results, which are then provided to the user.

[0177] This invention is a system that allows salespeople and store clerks to analyze conversations with customers in real time and generate and present appropriate responses during business negotiations and customer service. This system is realized by combining technologies using speech recognition, natural language processing, and generative AI models.

[0178] Terminal

[0179] The device first collects real-time audio data during business negotiations and meetings using smart glasses or smartphones, and then transcribes the collected audio data in real time using a speech recognition API.

[0180] server

[0181] The server receives the transcription data sent from the device. The received data is analyzed using natural language processing technology to evaluate the customer's sentiment and reactions. This analysis is performed using TextBlob or a similar tool. Based on the analysis results, the server uses a generative AI model to generate an appropriate talk script. The generative AI model typically uses the OpenAI API.

[0182] Users (salespeople and store clerks)

[0183] The user receives the talk script from the device in real time and immediately applies it to the conversation with the customer. For example, if the user is using smart glasses, the talk script is displayed visually. This allows the user to provide appropriate responses without interrupting the flow of the conversation.

[0184] Specific examples

[0185] 1. Audio data collection and transcription

[0186] If a customer says, "I'd like to know more about this product" during a sales negotiation, the device (smart glasses or smartphone) will collect the voice in real time and transcribe it using a speech recognition API.

[0187] 2. Analysis of the transcription data

[0188] The device sends the transcribed text data to the server, which analyzes it using TextBlob and determines that more information is required.

[0189] 3. Generate a talk script

[0190] The server uses the generative AI model to generate a talk script that says, "This product uses the latest technology and is particularly energy-efficient."

[0191] 4. Presenting the talk script

[0192] The generated talk script is sent to the terminal and displayed on the user's (salesperson or store clerk's) smart glasses. The user can use it as a reference to provide appropriate explanations to the customer.

[0193] Prompt Sentence Examples

[0194] An example of a prompt sentence input to the generative AI model is as follows:

[0195] "Customer question: I'd like to know more about this product. Good sales associate response:"

[0196] In this way, the present invention provides an effective means for salespeople and store clerks to respond appropriately in real time during business negotiations and customer service.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] The device collects voice data in real time during business negotiations or customer interactions. The collected voice data is then transcribed using a voice recognition API. The input for this step is the voice data from business negotiations or customer interactions, and the output is text data. This step includes the specific operation of calling the voice recognition API and converting voice to text.

[0200] Step 2:

[0201] The terminal sequentially transmits the transcribed text data to the server. The input of this step is the text data generated by speech recognition, and the output is the transcribed data transmitted to the server. This step includes specific operations of sequentially buffering and transmitting the text data to the server.

[0202] Step 3:

[0203] The server receives the transcription data sent from the terminal. The input of this step is the transcription data sent from the terminal, and the output is the received transcription data. The specific operation of the server receiving data over the network is included.

[0204] Step 4:

[0205] The server applies natural language processing techniques to the received transcription data to analyze the customer's emotions and reactions. The input of this step is the received transcription data, and the output is the analysis results (customer's emotions and reactions). Specifically, it uses TextBlob to analyze the customer's utterances and perform emotion evaluation.

[0206] Step 5:

[0207] The server generates an appropriate talk script based on the analysis results. The input to this step is the analysis results, and the output is the generated talk script. This step includes specific operations to generate an appropriate response using a generative AI model. The prompt sentence is input into the generative AI model to generate a talk script.

[0208] Step 6:

[0209] The server sends the generated talk script to the terminal. The input of this step is the generated talk script, and the output is the talk script sent to the terminal. This step includes a specific operation of sending the talk script to the terminal over the network.

[0210] Step 7:

[0211] The terminal presents the received talk script to the user (salesperson or store clerk). The input of this step is the talk script sent from the server, and the output is the talk script presented to the user. This step includes the specific operation of displaying the talk script on the screen of smart glasses or a smartphone.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] The present invention provides a system that enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system includes a means for transcribing conference audio data in real time, analyzing the transcript to evaluate customer emotions and reactions, and generating an appropriate talk script based on the results, which is then presented to the user. The system also includes an emotion engine that recognizes the user's emotions, enabling the generation of a talk script that takes the user's emotions into consideration.

[0214] System Configuration

[0215] 1. Terminal

[0216] The device records the audio of the meeting in real time.

[0217] The device sends the recorded audio to a speech recognition API, which transcribes it in real time.

[0218] The device buffers the transcription data sequentially and sends it to the server.

[0219] The device uses an emotion engine to analyze the user's facial expressions and voice tone and recognize their emotions.

[0220] 2. Server

[0221] The server receives the transcription data sent from the terminal.

[0222] The server uses natural language processing techniques to analyze the transcription data.

[0223] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[0224] It also integrates emotional data obtained from the user's emotion engine and dynamically updates the talk script.

[0225] The server sends the generated talk script to the terminal.

[0226] 3. User (Salesperson)

[0227] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[0228] The user's emotions are recognized and the data is used for analysis by the emotion engine.

[0229] Overview of program processing

[0230] 1. Acquiring and transcribing audio data

[0231] The device collects voice data during the meeting and converts it into text in real time using a speech recognition API.

[0232] 2. Sending the transcription data

[0233] The device sequentially buffers the generated transcription data and transmits it to the server at regular time-sliding intervals.

[0234] 3. Analysis of Transcription Data

[0235] The server receives the transcription data sent from the device and applies natural language processing techniques to analyze the text.

[0236] 4. User Emotion Recognition by Emotion Engine

[0237] The device uses an emotion engine to analyze the user's facial expressions and voice tone during the meeting and generate the user's emotion data in real time.

[0238] The terminal transmits the user's emotion data to the server.

[0239] 5. Evaluating emotions and reactions

[0240] The server integrates the analysis results with the user's emotional data to evaluate the customer's emotions and reactions.

[0241] For example, when a customer makes a negative comment such as "This product is too expensive," the emotion engine recognizes the user's impatience and confusion.

[0242] 6. Generating a Talk Script

[0243] The server generates an appropriate talk script based on the evaluation results, for example, a script that explains to the user, "Price is an important point, but our products offer excellent long-term cost performance."

[0244] 7. Sending the talk script

[0245] The server transmits the generated talk script to the terminal.

[0246] The terminal presents the talk script to the user visually or audibly.

[0247] Specific examples

[0248] Initial Setup and Starting a Meeting

[0249] The user starts the system and sets "concluding a contract for a new product" as the goal of the meeting.

[0250] The device activates its voice recognition and emotion engine and begins recording the meeting audio.

[0251] Real-time transcription and analysis

[0252] A customer says, "I think the price of this product is too high."

[0253] The device transcribes this speech in real time and sends the text data to a server.

[0254] The server analyzes the text data and evaluates the customer's sentiment as negative.

[0255] User Emotion Recognition

[0256] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions.

[0257] The terminal transmits the user's emotion data to the server.

[0258] Generate and present talk scripts

[0259] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[0260] The server sends the generated script to the terminal, which displays it to the user.

[0261] User response

[0262] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[0263] In this way, the present invention provides a system that supports the progress of a conference in real time and helps users take appropriate action while taking into account their emotions.

[0264] The processing flow will be explained below.

[0265] Step 1:

[0266] The user starts the system and sets the goal and main agenda of the meeting. For example, the user can enter "concluding a contract for a new product" as the goal.

[0267] Step 2:

[0268] The device will turn on its voice recognition and emotion engine and begin recording the meeting audio in real time.

[0269] Step 3:

[0270] The device sends voice data to the speech recognition API in real time and transcribes it. The transcribed text data is then buffered sequentially.

[0271] Step 4:

[0272] The device sends buffered transcription data to the server at regular time-sliding intervals.

[0273] Step 5:

[0274] The server receives the transcription data sent from the device and places it in an analysis queue.

[0275] Step 6:

[0276] The server uses natural language processing technology to analyze the received transcription data, for example, analyzing a statement such as "I think the price of this product is too high" to recognize a negative reaction to the price.

[0277] Step 7:

[0278] The device uses the user's facial recognition camera and microphone to analyze the user's facial expressions and voice tone using an emotion engine, generating the user's emotion data in real time.

[0279] Step 8:

[0280] The user's emotion data generated by the terminal is transmitted to the server.

[0281] Step 9:

[0282] The server combines the analysis results of the transcription data with the user's emotional data to evaluate the customer's emotions and reactions. For example, it may evaluate a comment such as "it's too expensive" as negative and determine that the user's emotions are impatience or confusion.

[0283] Step 10:

[0284] The server generates an appropriate talk script based on the evaluation results, for example, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[0285] Step 11:

[0286] The server sends the generated talk script to the terminal.

[0287] Step 12:

[0288] The terminal presents the generated talk script to the user visually or audibly.

[0289] Step 13:

[0290] The user refers to the provided talk script and replies, "Price is important, but this product offers excellent long-term cost performance," and continues the conversation.

[0291] Step 14:

[0292] The device records the audio again and repeats the process from step 3 onwards, generating new transcripts sequentially and sending them to the server, where the process of analysis, evaluation and script generation continues.

[0293] In this way, this system supports the progress of meetings in real time and enables appropriate responses to be made taking into account the feelings of both the user and the customer.

[0294] Example 2

[0295] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0296] Conventional meeting support systems were unable to transcribe voice data in real time or perform subsequent analysis quickly and accurately. Furthermore, it was difficult to generate appropriate talk scripts that took into account the customer's emotions and reactions, placing a heavy burden on sales representatives. Furthermore, there were insufficient means to dynamically respond to the progress of the meeting and the user's emotions, making it difficult to conduct effective sales activities.

[0297] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring conference audio in real time and converting it into text data, means for transmitting the text data to the server at regular intervals, means for analyzing the text data using natural language processing technology, means for analyzing the emotions of the user and the customer in real time using an emotion analysis engine, means for integrating the emotion data of the customer and the user and evaluating the customer's reaction, means for generating a talk script based on the evaluation result, and means for presenting the generated talk script to the user. This enables real-time analysis of audio data and rapid generation of a talk script according to the customer's emotion.

[0298] "Means for capturing conference audio in real time and converting it into text data" refers to a device or system that includes technology for instantly capturing audio generated during a conference as digital data and converting that audio data into text data.

[0299] The "means for transmitting character data to a server at regular intervals" refers to a device or system for transmitting the generated character data to a server at appropriate timing.

[0300] "Means for analyzing text data using natural language processing techniques" refers to a device or system that uses natural language processing algorithms and techniques to perform text analysis on acquired text data.

[0301] "Means for analyzing user and customer emotions in real time using an emotion analysis engine" refers to a device or system that includes technology for analyzing the voice tone and facial expressions of users and customers and evaluating their emotions in real time.

[0302] "Means for integrating emotional data of customers and users and evaluating customer reactions" refers to a device or system that includes technology for integrating emotional data obtained from users and customers and evaluating customer reactions based on the results.

[0303] The "means for generating a talk script based on the evaluation results" refers to a device or system including a technology for generating a talk script including appropriate conversation content based on the customer's reaction evaluation results.

[0304] The "means for presenting the generated talk script to the user" refers to a device or system that includes a technique for visually or audibly presenting the generated talk script to the user.

[0305] A "generative AI model" is an artificial intelligence algorithm trained using large datasets, and is a model for generating and analyzing text data.

[0306] A "prompt" is an instruction entered into a generative AI model, and is text used to control the AI's behavior and generation.

[0307] The present invention provides a system for supporting salespeople in responding to customers promptly and appropriately in meetings and business negotiations. Specific embodiments for carrying out the present invention will be described below.

[0308] This system is broadly composed of two main components: a terminal and a server.

[0309] Device configuration

[0310] The terminal is a device used in a meeting or business negotiation, and specifically includes a microphone, a camera, a voice recognition API, and an emotion analysis API.

[0311] 1. Acquire audio data:

[0312] The device uses a built-in microphone to collect audio data during the meeting, and audio capture is performed in real time.

[0313] 2. Audio data conversion:

[0314] The device converts the collected voice data into text data using the Google Speech-to-Text API or the IBM Watson (registered trademark) Speech to Text API.

[0315] 3. Emotion recognition:

[0316] The device uses a camera to capture the user's facial expressions and a microphone to capture voice tones. This data is then analyzed using emotion analysis APIs such as Microsoft Azure Emotion API and Amazon Rekognition to generate emotion data in real time.

[0317] 4. Data transmission:

[0318] The terminal buffers the generated character data and emotion data at regular intervals and transmits them to the server.

[0319] Server Configuration

[0320] The server is a system that receives data sent from the device, analyzes it, and generates a talk script, using natural language processing technology and generative AI models.

[0321] 1. Receiving data:

[0322] The server receives the character data and emotion data sent from the terminal.

[0323] 2. Data Analysis:

[0324] The server uses natural language processing technology (for example, Python's NLTK library or SpaCy) to analyze the received text data and evaluate the intent and sentiment of the customer's comments.

[0325] 3. Emotional data integration:

[0326] The server integrates the received emotional data and evaluates the customer's reaction comprehensively.

[0327] 4. Generate the talk script:

[0328] The server uses a generative AI model (e.g., OpenAI's GPT-3) based on the analysis results to generate an appropriate conversation script, which provides the optimal response method for the user.

[0329] 5. Sending the talk script:

[0330] The server transmits the generated talk script to the terminal, which then presents the talk script to the user visually or audibly.

[0331] Specific examples

[0332] Initial Setup and Starting a Meeting

[0333] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product." The device then activates its voice recognition function and emotion engine and begins recording the audio of the meeting.

[0334] Real-time transcription and analysis example

[0335] A customer says, "I think the price of this product is too high." The device transcribes this speech in real time and sends the text data to the server. The server analyzes the text data and evaluates the customer's sentiment as negative.

[0336] Example of user emotion recognition

[0337] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions, and then transmits the user's emotion data to the server.

[0338] Example of generating and presenting a talk script

[0339] The server generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance." The server sends the generated script to the terminal, which then displays it to the user.

[0340] Example of user response

[0341] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[0342] Prompt Sentence Examples

[0343] "Provide a script to alleviate the feelings of impatience identified by the emotion engine."

[0344] "How can I generate a talk script to respond to negative comments from customers?"

[0345] This system enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, enabling them to conduct effective sales activities.

[0346] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0347] Step 1: Acquire and transcribe audio data

[0348] Input: A user starts a conference and a microphone collects audio.

[0349] How it works: The device captures real-time audio during meetings, which is then sent to the Google Speech-to-Text API or IBM Watson Speech to Text API for conversion into text data.

[0350] Data processing: The audio data is converted into text data. Specifically, the API analyzes the audio waveform and generates the corresponding text.

[0351] Output: Text data (e.g., "This product is too expensive").

[0352] Step 2: Sending character data

[0353] Input: The character data generated in step 1.

[0354] Operation: The terminal buffers the generated character data at a fixed time-slide interval and transmits it to the server sequentially.

[0355] Data processing: Data is accumulated through buffering and converted into a format that can be sent in fixed amounts.

[0356] Output: Character data sent to the server.

[0357] Step 3: Parsing character data

[0358] Input: Character data sent to the server.

[0359] How it works: The server analyzes text data using Python's NLTK library, SpaCy, etc. Specifically, it performs semantic and sentiment analysis of the text.

[0360] Data processing: Grammatical analysis, keyword extraction, and sentiment analysis are performed on the text data.

[0361] Output: Analysis results (e.g., "negative sentiment", "price mentions").

[0362] Step 4: Emotion recognition and data transmission

[0363] Input: User's voice tone and facial expression data during the meeting.

[0364] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice tone, and this data is analyzed in real time using tools like the Microsoft Azure Emotion API and Amazon Rekognition.

[0365] Data processing: The captured voice tone and facial expression data are converted into emotion data through emotion analysis algorithms.

[0366] Output: Generated user emotion data (e.g., "impatience").

[0367] Step 5: Assess your emotions and reactions

[0368] Input: Analysis results of text data and emotion data.

[0369] Operation: The server integrates the results of the character data analysis with the user's emotional data to evaluate the customer's reaction. Specifically, it compares both sets of emotional data and performs a comprehensive emotional evaluation.

[0370] Data processing: By integrating text data and emotion data, more detailed emotion evaluation is performed.

[0371] Output: Customer reaction evaluation results (e.g., "Customers reacted negatively to the price").

[0372] Step 6: Generate the talk script

[0373] Input: Customer response evaluation results.

[0374] Operation: The server uses the generative AI model to generate an appropriate talk script based on the evaluation results. The server inputs the prompt sentence into the generative AI model and obtains the script.

[0375] Data processing: Generative AI models generate natural-sounding conversation scripts based on context.

[0376] Output: Generated talk script (e.g., "Price is important, but this product offers great long-term value for money").

[0377] Step 7: Send and present your talk script

[0378] Input: The generated talk script.

[0379] Operation: The server sends the generated talk script to the terminal, which receives it and presents it to the user visually or audibly.

[0380] Data processing: The talk script is converted into a format that is easy for the user to view or to use in audio.

[0381] Output: The talk script presented to the user (e.g., the script displayed on the device screen).

[0382] Through this detailed step-by-step process, the present invention helps users respond quickly and appropriately during meetings and business negotiations with customers.

[0383] (Application example 2)

[0384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] It is important for sales representatives to respond quickly and appropriately to customer emotions and reactions during meetings and negotiations. However, it is difficult to understand the content of the conversation in real time, instantly grasp the customer's emotions, and provide an appropriate dialogue script that matches them. There is a need to solve this problem and significantly improve the efficiency and effectiveness of sales activities. Furthermore, to achieve real-time support, it is necessary to operate an integrated system on a device worn by the sales representative.

[0386] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0387] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for displaying the generated talk script in real time on a device worn by a sales representative. This enables the sales representative to respond appropriately on the spot during a conversation with a customer, thereby increasing the success rate of meetings and business negotiations.

[0388] "Conference audio data" refers to all audio information spoken during the conference, including conversations between the sales representative and the customer.

[0389] "Real-time transcription means" refers to a technical means for instantly converting collected audio data into text data.

[0390] "Transcription data" means audio data converted into text format.

[0391] "Means for analyzing the transcript data" refers to the technical means for analyzing the converted transcript data to understand and classify its content.

[0392] "Means for assessing emotions and reactions" refers to technological means for determining customer emotions and reactions based on analyzed transcript data.

[0393] A "talk script" is a document that provides instructions and examples of conversations that a sales representative can use to respond appropriately to a customer.

[0394] "Generative means" refers to technological means that use engines or algorithms to create new information or data for a specific purpose.

[0395] "Means for presenting" refers to a technical means for displaying or notifying the generated talk script to the sales representative.

[0396] "Means for displaying on a device in real time" refers to a technical means for instantly providing the generated talk script on the display screen of a device (such as smart glasses) worn by a sales representative.

[0397] (Embodiment of the Invention)

[0398] The present invention provides a system that enables salespeople to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system transcribes voice data in real time, analyzes the transcript to evaluate the customer's emotions and reactions, generates an appropriate talk script based on the analysis, and displays it in real time on a device worn by the salesperson.

[0399] System Configuration

[0400] 1. Terminal

[0401] The device (e.g., smart glasses) performs the following operations:

[0402] Audio data collection: The built-in microphone of the smart glasses is used to record the audio of the meeting in real time.

[0403] Transcription: Recorded audio data is transcribed in real time using the Google Cloud Speech-to-Text API.

[0404] Emotion recognition: The smart glasses' built-in camera captures the customer's facial expressions and uses the Microsoft Azure Face API to analyze their emotions in real time.

[0405] Data transmission: The buffered transcription data and emotion data are sent to the server.

[0406] 2. Server

[0407] The server performs the following operations:

[0408] Data reception: Receives transcription data and emotion data sent from the device.

[0409] Natural Language Processing: Analyze the received transcription data using natural language processing techniques (e.g., NLTK with Sentiment Analysis).

[0410] Sentiment assessment: Evaluate customer emotions and reactions based on the analysis results.

[0411] Talk script generation: Use a generative AI model such as OpenAI's GPT-4 (registered trademark) to generate an appropriate talk script based on the evaluation results.

[0412] Send data: Send the generated talk script to the device.

[0413] 3. User (Salesperson)

[0414] Salespeople use the system as follows:

[0415] Check the talk script: Refer to the talk script displayed on the terminal and proceed with the conversation with the customer.

[0416] Real-time response: Based on the generated conversation script, appropriate responses are provided according to the customer's emotions.

[0417] Specific examples

[0418] Initial Setup and Conference Start Example

[0419] A sales representative puts on the smart glasses and starts the system. The purpose of the meeting is set to "conclude a contract for a new product."

[0420] The smart glasses activate their voice recognition and emotion recognition engines and begin recording and capturing the meeting audio and customer facial expressions in real time.

[0421] Transcription and analysis examples

[0422] A customer says, "I think this product is priced too high."

[0423] The smart glasses transcribe this speech in real time and send the text data to a server.

[0424] The server analyzes the text data and evaluates the customer's sentiment as negative.

[0425] Examples of prompt statements

[0426] Audio Transcription: 'I think the price of this product is too high.'

[0427] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[0428] Generate the appropriate talk script.

[0429] As described above, the present invention provides a system that supports the progress of a conference in real time and supports the responses of sales representatives.

[0430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0431] Step 1:

[0432] Audio data collection

[0433] The device (smart glasses) uses a built-in microphone to record the audio of the meeting in real time.

[0434] Input: Audio data during the meeting

[0435] Output: Recorded audio data

[0436] Specifically, the system records the conversation between the sales representative and the customer in high quality and temporarily stores it in local memory for post-processing.

[0437] Step 2:

[0438] Transcription

[0439] The device transcribes the recorded audio data in real time using the Google Cloud Speech-to-Text API.

[0440] Input: Recorded audio data

[0441] Output: Real-time transcribed text data

[0442] Specifically, audio data is sent to the API, and the resulting text data is buffered and recorded sequentially.

[0443] Step 3:

[0444] emotion recognition

[0445] The device captures the customer's facial expressions using its built-in camera, and analyzes this video data in real time using the Microsoft Azure Face API to assess the customer's emotions.

[0446] Input: Customer facial expression video data

[0447] Output: Real-time analyzed emotion data

[0448] Specifically, the device's camera captures video of the customer at regular intervals, sends the video to an API, and obtains emotion recognition results.

[0449] Step 4:

[0450] Sending data

[0451] The terminal transmits the buffered transcription data and real-time emotion data to the server at regular time-sliding intervals.

[0452] Input: Transcription text data, emotion data

[0453] Output: Send data to the server

[0454] Specifically, the buffered data is collected into packets at regular intervals and sent to a specified address on the server.

[0455] Step 5:

[0456] Natural Language Processing

[0457] The server analyzes the received transcription data using natural language processing technology (e.g., NLTK with Sentiment Analysis) to understand and classify the content of the conversation.

[0458] Input: Transcribed text data sent to the server

[0459] Output: Analysis data after natural language processing

[0460] Specifically, the text data is tokenized, and the context and sentiment of each token is analyzed to determine the meaning and sentiment of the customer's words.

[0461] Step 6:

[0462] Emotional evaluation

[0463] The server evaluates the customer's emotions and reactions based on the analysis results, which also incorporates the received emotional data.

[0464] Input: Analysis data after natural language processing, emotion data

[0465] Output: Evaluated customer sentiment and reaction data

[0466] Specifically, analytical data and emotional data are combined to determine and evaluate the actual emotions felt by the customer.

[0467] Step 7:

[0468] Generate a talk script

[0469] The server uses a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[0470] Input: Evaluated customer sentiment and reaction data

[0471] Output: Generated talk script

[0472] Specifically, the prompt sentence is input into the generative AI model, and the obtained talk script is organized and saved.

[0473] Specific prompt examples:

[0474] Audio Transcription: 'I think the price of this product is too high.'

[0475] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[0476] Generate the appropriate talk script.

[0477] Step 8:

[0478] Sending data

[0479] The server transmits the generated talk script to the terminal.

[0480] Input: Generated talk script

[0481] Output: Sending data to the terminal

[0482] Specifically, the generated talk script is sent to the specified address of the terminal.

[0483] Step 9:

[0484] Show talk script

[0485] The terminal displays the generated talk script in real time within the field of view of the salesperson.

[0486] Input: Talk script received from the server

[0487] Output: Talk script displayed to sales representatives

[0488] Specifically, the talk script will be visually displayed on the device display, allowing sales representatives to quickly refer to it.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0492] [Second embodiment]

[0493] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0494] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0496] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0500] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0503] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0504] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0505] The present invention provides a system for enabling a sales representative to respond quickly and appropriately in meetings and business negotiations with customers. The system includes the following components.

[0506] System Configuration

[0507] 1. Terminal

[0508] The device records the audio of the meeting in real time.

[0509] The device calls a speech recognition API to transcribe the recorded audio in real time.

[0510] The device sequentially buffers the generated transcription data and sends it to the server.

[0511] 2. Server

[0512] The server receives the transcription data sent from the terminal.

[0513] The server uses natural language processing technology to analyze the transcription data.

[0514] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[0515] The server sends the generated talk script to the terminal.

[0516] 3. User (Salesperson)

[0517] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[0518] Overview of program processing

[0519] 1. Acquiring and transcribing audio data

[0520] The device collects audio data during the meeting and transcribes it in real time using a speech recognition API.

[0521] The device buffers the transcription data sequentially and transmits it to the server at regular time-sliding intervals.

[0522] 2. Analysis of the transcription data

[0523] The server receives the transcription data sent from the terminal.

[0524] The server applies natural language processing techniques to the received data and analyzes the text.

[0525] Example: Analyze the statement "I think this product is too expensive" to recognize negative evaluations of price.

[0526] 3. Evaluating emotions and reactions

[0527] The server evaluates the customer's emotions and reactions based on the analysis results.

[0528] Example: The statement "it's too expensive" is judged to indicate a negative sentiment.

[0529] 4. Generate a talk script

[0530] The server generates an appropriate talk script based on the evaluation result.

[0531] Example: Generates a script that says, "Price is certainly an important factor, but our products offer excellent long-term value for money."

[0532] 5. Send and present the talk script

[0533] The server sends the generated talk script to the terminal.

[0534] The terminal presents the talk script to the user visually or audibly.

[0535] Specific examples

[0536] Initial Setup and Starting a Meeting

[0537] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product."

[0538] The device will turn on the voice recognition function and begin recording the meeting audio.

[0539] Real-time transcription and analysis

[0540] A customer says, "I think the price of this product is too high."

[0541] The device transcribes this speech in real time and sends the text data to a server.

[0542] The server analyzes the text data and evaluates the customer's sentiment as negative.

[0543] Generate and present talk scripts

[0544] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[0545] The server sends the generated script to the terminal, which displays it to the user.

[0546] User response

[0547] Using the script provided, the user responds, "Price is certainly an important point, but our products offer excellent long-term cost performance," and continues the conversation with the customer.

[0548] In this way, the present invention provides an effective tool to assist salespeople in responding quickly and appropriately during a meeting.

[0549] The processing flow will be explained below.

[0550] Step 1:

[0551] The user starts the system and sets the goal and main agenda of the meeting. For example, the user enters "concluding a contract for a new product" as the goal.

[0552] Step 2:

[0553] The device will turn on the voice recognition function and begin recording the meeting audio in real time.

[0554] Step 3:

[0555] The device sends the recorded voice data to the speech recognition API and converts it into text data in real time. The converted text data is buffered sequentially.

[0556] Step 4:

[0557] The device sends buffered transcription data to the server at regular time-sliding intervals.

[0558] Step 5:

[0559] The server receives the transcription data sent from the device and places the received data in a queue for analysis.

[0560] Step 6:

[0561] The server uses natural language processing technology to analyze the transcription data, for example, analyzing a customer statement in the text, "I think the price of this product is too high," to recognize a negative reaction to the price.

[0562] Step 7:

[0563] The server evaluates the customer's emotions and reactions based on the analysis results. Comments such as "too expensive" are evaluated negatively, and the server determines that the enthusiasm for the meeting is declining.

[0564] Step 8:

[0565] Based on the evaluation results, the server generates an appropriate talk script, such as, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[0566] Step 9:

[0567] The server sends the generated talk script to the terminal, which is then used to continue the conference.

[0568] Step 10:

[0569] The terminal displays the generated talk script to the user, who then uses this script to speak to the customer.

[0570] Step 11:

[0571] The user responds, "Price is certainly an important factor, but our products offer excellent long-term value for money," and continues the conversation.

[0572] Step 12:

[0573] The device records the speech again and repeats the process from step 3. Each time new transcription data is generated, it is sent sequentially to the server, where the process of analysis, evaluation, and script generation continues.

[0574] In this way, the system helps manage meetings in real time, enabling salespeople to effectively serve customers.

[0575] Example 1

[0576] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0577] The lack of effective support tools that enable sales representatives to respond quickly and appropriately during meetings and negotiations with customers is an issue. Specifically, the lack of real-time feedback on customer comments and the provision of appropriate conversation scripts can reduce the success rate of sales negotiations.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0579] In this invention, the server includes means for acquiring conference audio data in real time using a voice input device, means for transcribing in real time using a voice recognition API, means for sequentially buffering the transcribed data and transmitting it to the server at regular time slide intervals, means for receiving the transcribed data and analyzing it using natural language processing technology, means for evaluating the customer's emotions and reactions based on the analysis results, means for generating an appropriate talk script using a generative AI model based on the evaluation results, and means for presenting the generated talk script to the user. This enables sales representatives to respond quickly and appropriately to customer comments and emotions during meetings and lead sales negotiations to success.

[0580] The "voice input device" is hardware for acquiring conference voice data in real time.

[0581] A "voice recognition API" is a software interface for converting acquired voice data into text data.

[0582] "Buffering" is a process of temporarily storing transcription data and sending it to a server at regular time-slide intervals.

[0583] "Natural language processing technology" refers to algorithms and technologies for analyzing text data and understanding meaning and emotions.

[0584] A "generative AI model" is an artificial intelligence algorithm that generates an appropriate talk script based on the analysis results.

[0585] A "talk script" is a conversation script that sales representatives can use as a reference, generated based on the customer's emotions and reactions.

[0586] "Sequential" means performing a series of data or processes one step at a time in order.

[0587] The present invention provides a system that enables salespeople to respond quickly and appropriately in meetings and business negotiations with customers. This system operates in cooperation with terminals, a server, and users.

[0588] System Configuration

[0589] 1. Terminal

[0590] The terminal uses a voice input device to capture the audio data of the meeting in real time. The hardware used here is a regular smartphone or tablet.

[0591] The device uses speech recognition APIs such as the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services to transcribe in real time.

[0592] The terminal has a communication function for sequentially buffering the transcription data and transmitting it to the server at regular time-sliding intervals.

[0593] Examples:

[0594] When a user starts a meeting, they launch the application on their device and turn on the speech recognition function. Each utterance made during the meeting is transcribed in real time and continuously sent to the server.

[0595] Example prompt sentence:

[0596] "Transcribe meeting audio in real time."

[0597] 2. Server

[0598] The server receives the transcription data sent from the device. A regular web server is used here, and an RDBMS such as MySQL or PostgreSQL is used as the database.

[0599] The server analyzes the received text data using a Python natural language processing library (e.g., NLTK or SpaCy).

[0600] The server evaluates the customer's emotions and reactions based on the analysis results. For example, it determines that a comment such as "too expensive" indicates a negative evaluation of the price.

[0601] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate talk script.

[0602] Examples:

[0603] If a customer says during a meeting, "I think the price of this product is too high," the server analyzes the comment in real time, detects the negative sentiment of "high price," and then generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance."

[0604] Example prompt sentence:

[0605] "Generate a talking script to use when a customer has a negative comment about the price."

[0606] 3. User (Salesperson)

[0607] The user can proceed with the conversation by referring to the talk script provided by the device, which can either display the script visually or read it aloud.

[0608] Examples:

[0609] During the meeting, the user responds to the customer using the script displayed on the terminal, which reads, "Price is certainly an important point. However, our products offer excellent long-term cost performance."

[0610] Example prompt sentence:

[0611] "Show the generated talk script to the user."

[0612] With the above configuration, the system of the present invention enables salespeople to respond quickly and appropriately to customer comments and feelings during a meeting, leading to successful business negotiations.

[0613] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0614] Step 1:

[0615] The terminal acquires the audio data of the conference using an audio input device. The input is the audio data of the conference, and the output is the acquired audio data. Specifically, the user launches an application on the terminal and turns on the voice recognition function. The terminal records the conference audio through the microphone.

[0616] Step 2:

[0617] The device transcribes the acquired audio data in real time using a speech recognition API. The input is the audio data from step 1, and the output is the transcribed text data. This operation uses the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services. Specifically, the device sends the audio data to the API and receives the text data returned by the API.

[0618] Step 3:

[0619] The device buffers the transcription data sequentially and sends it to the server at regular time-sliding intervals. The input is the transcription data from step 2, and the output is the text data sent to the server. Specifically, the device buffers the text data obtained every 5 seconds and sends it to the server via an HTTP POST request.

[0620] Step 4:

[0621] The server receives the transcription data sent from the device. The input is the text data sent in step 3, and the output is the received transcription data. Specifically, the server receives the HTTP request and writes the data to a database (MySQL or PostgreSQL).

[0622] Step 5:

[0623] The server analyzes the received transcription data using natural language processing technology. The input is the data received in step 4, and the output is the analysis results. This operation uses Python's NLTK and SpaCy libraries. Specifically, the server tokenizes the text data and performs sentiment analysis.

[0624] Step 6:

[0625] The server evaluates the customer's emotions and reactions based on the analysis results. The input is the analysis result from step 5, and the output is the evaluation result. Specifically, the server checks keywords such as "high" and "negative" against a dictionary and sets a negative flag.

[0626] Step 7:

[0627] Based on the evaluation results, the server uses a generative AI model to generate an appropriate talk script. The input is the evaluation result from step 6, and the output is the generated talk script. This operation uses a generative AI model such as OpenAI's GPT-3. Specifically, the server inputs a prompt statement (e.g., "Please generate a talk script that can be used when making negative comments about prices") into the AI ​​model and receives the script generated by the AI ​​model.

[0628] Step 8:

[0629] The server sends the generated talk script to the terminal. The input is the talk script from step 7, and the output is the script sent to the terminal. Specifically, the server sends the talk script to the terminal using an HTTP POST request.

[0630] Step 9:

[0631] The terminal presents the received talk script to the user. The input is the talk script from step 8, and the output is the script presented to the user. Specifically, the terminal displays the script in the application's display area or provides a function to read it aloud.

[0632] (Application example 1)

[0633] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0634] In traditional sales negotiations and customer service, it was difficult for salespeople and store clerks to accurately grasp what customers were saying and how they were feeling, and then respond appropriately immediately. Furthermore, because conversations were recorded and analyzed manually, real-time support was difficult, leading to inconsistencies in the quality of customer service. This resulted in issues such as lower customer satisfaction and a lower success rate for sales negotiations.

[0635] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0636] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for collecting audio data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time, thereby enabling salespeople or store clerks to receive appropriate feedback and responses in real time during conversations with customers.

[0637] "Means for transcribing meeting audio data in real time" is a technology that converts audio data collected during meetings or business negotiations into text data in real time.

[0638] "Means of analyzing transcribed data and evaluating customer emotions and reactions" refers to a technology that analyzes transcribed data using natural language processing technology and evaluates customer emotions and reactions.

[0639] "Means for generating appropriate talk scripts based on customers' emotions and reactions" refers to technology that automatically generates appropriate response content based on customers' emotions and reactions obtained through analysis.

[0640] The "means for presenting the generated talk script to the user" is a technique for immediately providing the generated response content to the user visually or audibly.

[0641] "Means for collecting voice data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time" refers to technology that collects voice data in real time during business negotiations or customer interactions, analyzes the data, and instantly generates feedback and responses based on the results, which are then provided to the user.

[0642] This invention is a system that allows salespeople and store clerks to analyze conversations with customers in real time and generate and present appropriate responses during business negotiations and customer service. This system is realized by combining technologies using speech recognition, natural language processing, and generative AI models.

[0643] Terminal

[0644] The device first collects real-time audio data during business negotiations and meetings using smart glasses or smartphones, and then transcribes the collected audio data in real time using a speech recognition API.

[0645] server

[0646] The server receives the transcription data sent from the device. The received data is analyzed using natural language processing technology to evaluate the customer's sentiment and reactions. This analysis is performed using TextBlob or a similar tool. Based on the analysis results, the server uses a generative AI model to generate an appropriate talk script. The generative AI model typically uses the OpenAI API.

[0647] Users (salespeople and store clerks)

[0648] The user receives the talk script from the device in real time and immediately applies it to the conversation with the customer. For example, if the user is using smart glasses, the talk script is displayed visually. This allows the user to provide appropriate responses without interrupting the flow of the conversation.

[0649] Specific examples

[0650] 1. Audio data collection and transcription

[0651] If a customer says, "I'd like to know more about this product" during a sales negotiation, the device (smart glasses or smartphone) will collect the voice in real time and transcribe it using a speech recognition API.

[0652] 2. Analysis of the transcription data

[0653] The device sends the transcribed text data to the server, which analyzes it using TextBlob and determines that more information is required.

[0654] 3. Generate a talk script

[0655] The server uses the generative AI model to generate a talk script that says, "This product uses the latest technology and is particularly energy-efficient."

[0656] 4. Presenting the talk script

[0657] The generated talk script is sent to the terminal and displayed on the user's (salesperson or store clerk's) smart glasses. The user can use it as a reference to provide appropriate explanations to the customer.

[0658] Prompt Sentence Examples

[0659] An example of a prompt sentence input to the generative AI model is as follows:

[0660] "Customer question: I'd like to know more about this product. Good sales associate response:"

[0661] In this way, the present invention provides an effective means for salespeople and store clerks to respond appropriately in real time during business negotiations and customer service.

[0662] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0663] Step 1:

[0664] The device collects voice data in real time during business negotiations or customer interactions. The collected voice data is then transcribed using a voice recognition API. The input for this step is the voice data from business negotiations or customer interactions, and the output is text data. This step includes the specific operation of calling the voice recognition API and converting voice to text.

[0665] Step 2:

[0666] The terminal sequentially transmits the transcribed text data to the server. The input of this step is the text data generated by speech recognition, and the output is the transcribed data transmitted to the server. This step includes specific operations of sequentially buffering and transmitting the text data to the server.

[0667] Step 3:

[0668] The server receives the transcription data sent from the terminal. The input of this step is the transcription data sent from the terminal, and the output is the received transcription data. The specific operation of the server receiving data over the network is included.

[0669] Step 4:

[0670] The server applies natural language processing techniques to the received transcription data to analyze the customer's emotions and reactions. The input of this step is the received transcription data, and the output is the analysis results (customer's emotions and reactions). Specifically, it uses TextBlob to analyze the customer's utterances and perform emotion evaluation.

[0671] Step 5:

[0672] The server generates an appropriate talk script based on the analysis results. The input to this step is the analysis results, and the output is the generated talk script. This step includes specific operations to generate an appropriate response using a generative AI model. The prompt sentence is input into the generative AI model to generate a talk script.

[0673] Step 6:

[0674] The server sends the generated talk script to the terminal. The input of this step is the generated talk script, and the output is the talk script sent to the terminal. This step includes a specific operation of sending the talk script to the terminal over the network.

[0675] Step 7:

[0676] The terminal presents the received talk script to the user (salesperson or store clerk). The input of this step is the talk script sent from the server, and the output is the talk script presented to the user. This step includes the specific operation of displaying the talk script on the screen of smart glasses or a smartphone.

[0677] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0678] The present invention provides a system that enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system includes a means for transcribing conference audio data in real time, analyzing the transcript to evaluate customer emotions and reactions, and generating an appropriate talk script based on the results, which is then presented to the user. The system also includes an emotion engine that recognizes the user's emotions, enabling the generation of a talk script that takes the user's emotions into consideration.

[0679] System Configuration

[0680] 1. Terminal

[0681] The device records the audio of the meeting in real time.

[0682] The device sends the recorded audio to a speech recognition API, which transcribes it in real time.

[0683] The device buffers the transcription data sequentially and sends it to the server.

[0684] The device uses an emotion engine to analyze the user's facial expressions and voice tone and recognize their emotions.

[0685] 2. Server

[0686] The server receives the transcription data sent from the terminal.

[0687] The server uses natural language processing techniques to analyze the transcription data.

[0688] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[0689] It also integrates emotional data obtained from the user's emotion engine and dynamically updates the talk script.

[0690] The server sends the generated talk script to the terminal.

[0691] 3. User (Salesperson)

[0692] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[0693] The user's emotions are recognized and the data is used for analysis by the emotion engine.

[0694] Overview of program processing

[0695] 1. Acquiring and transcribing audio data

[0696] The device collects voice data during the meeting and converts it into text in real time using a speech recognition API.

[0697] 2. Sending the transcription data

[0698] The device sequentially buffers the generated transcription data and transmits it to the server at regular time-sliding intervals.

[0699] 3. Analysis of Transcription Data

[0700] The server receives the transcription data sent from the device and applies natural language processing techniques to analyze the text.

[0701] 4. User Emotion Recognition by Emotion Engine

[0702] The device uses an emotion engine to analyze the user's facial expressions and voice tone during the meeting and generate the user's emotion data in real time.

[0703] The terminal transmits the user's emotion data to the server.

[0704] 5. Evaluating emotions and reactions

[0705] The server integrates the analysis results with the user's emotional data to evaluate the customer's emotions and reactions.

[0706] For example, when a customer makes a negative comment such as "This product is too expensive," the emotion engine recognizes the user's impatience and confusion.

[0707] 6. Generating a Talk Script

[0708] The server generates an appropriate talk script based on the evaluation results, for example, a script that explains to the user, "Price is an important point, but our products offer excellent long-term cost performance."

[0709] 7. Sending the talk script

[0710] The server transmits the generated talk script to the terminal.

[0711] The terminal presents the talk script to the user visually or audibly.

[0712] Specific examples

[0713] Initial Setup and Starting a Meeting

[0714] The user starts the system and sets "concluding a contract for a new product" as the goal of the meeting.

[0715] The device activates its voice recognition and emotion engine and begins recording the meeting audio.

[0716] Real-time transcription and analysis

[0717] A customer says, "I think the price of this product is too high."

[0718] The device transcribes this speech in real time and sends the text data to a server.

[0719] The server analyzes the text data and evaluates the customer's sentiment as negative.

[0720] User Emotion Recognition

[0721] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions.

[0722] The terminal transmits the user's emotion data to the server.

[0723] Generate and present talk scripts

[0724] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[0725] The server sends the generated script to the terminal, which displays it to the user.

[0726] User response

[0727] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[0728] In this way, the present invention provides a system that supports the progress of a conference in real time and helps users take appropriate action while taking into account their emotions.

[0729] The processing flow will be explained below.

[0730] Step 1:

[0731] The user starts the system and sets the goal and main agenda of the meeting. For example, the user can enter "concluding a contract for a new product" as the goal.

[0732] Step 2:

[0733] The device will turn on its voice recognition and emotion engine and begin recording the meeting audio in real time.

[0734] Step 3:

[0735] The device sends voice data to the speech recognition API in real time and transcribes it. The transcribed text data is then buffered sequentially.

[0736] Step 4:

[0737] The device sends buffered transcription data to the server at regular time-sliding intervals.

[0738] Step 5:

[0739] The server receives the transcription data sent from the device and places it in an analysis queue.

[0740] Step 6:

[0741] The server uses natural language processing technology to analyze the received transcription data, for example, analyzing a statement such as "I think the price of this product is too high" to recognize a negative reaction to the price.

[0742] Step 7:

[0743] The device uses the user's facial recognition camera and microphone to analyze the user's facial expressions and voice tone using an emotion engine, generating the user's emotion data in real time.

[0744] Step 8:

[0745] The user's emotion data generated by the terminal is transmitted to the server.

[0746] Step 9:

[0747] The server combines the analysis results of the transcription data with the user's emotional data to evaluate the customer's emotions and reactions. For example, it may evaluate a comment such as "it's too expensive" as negative and determine that the user's emotions are impatience or confusion.

[0748] Step 10:

[0749] The server generates an appropriate talk script based on the evaluation results, for example, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[0750] Step 11:

[0751] The server sends the generated talk script to the terminal.

[0752] Step 12:

[0753] The terminal presents the generated talk script to the user visually or audibly.

[0754] Step 13:

[0755] The user refers to the provided talk script and replies, "Price is important, but this product offers excellent long-term cost performance," and continues the conversation.

[0756] Step 14:

[0757] The device records the audio again and repeats the process from step 3 onwards, generating new transcripts sequentially and sending them to the server, where the process of analysis, evaluation and script generation continues.

[0758] In this way, this system supports the progress of meetings in real time and enables appropriate responses to be made taking into account the feelings of both the user and the customer.

[0759] Example 2

[0760] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0761] Conventional meeting support systems were unable to transcribe voice data in real time or perform subsequent analysis quickly and accurately. Furthermore, it was difficult to generate appropriate talk scripts that took into account the customer's emotions and reactions, placing a heavy burden on sales representatives. Furthermore, there were insufficient means to dynamically respond to the progress of the meeting and the user's emotions, making it difficult to conduct effective sales activities.

[0762] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring conference audio in real time and converting it into text data, means for transmitting the text data to the server at regular intervals, means for analyzing the text data using natural language processing technology, means for analyzing the emotions of the user and the customer in real time using an emotion analysis engine, means for integrating the emotion data of the customer and the user and evaluating the customer's reaction, means for generating a talk script based on the evaluation result, and means for presenting the generated talk script to the user. This enables real-time analysis of audio data and rapid generation of a talk script according to the customer's emotion.

[0763] "Means for capturing conference audio in real time and converting it into text data" refers to a device or system that includes technology for instantly capturing audio generated during a conference as digital data and converting that audio data into text data.

[0764] The "means for transmitting character data to a server at regular intervals" refers to a device or system for transmitting the generated character data to a server at appropriate timing.

[0765] "Means for analyzing text data using natural language processing techniques" refers to a device or system that uses natural language processing algorithms and techniques to perform text analysis on acquired text data.

[0766] "Means for analyzing user and customer emotions in real time using an emotion analysis engine" refers to a device or system that includes technology for analyzing the voice tone and facial expressions of users and customers and evaluating their emotions in real time.

[0767] "Means for integrating emotional data of customers and users and evaluating customer reactions" refers to a device or system that includes technology for integrating emotional data obtained from users and customers and evaluating customer reactions based on the results.

[0768] The "means for generating a talk script based on the evaluation results" refers to a device or system including a technology for generating a talk script including appropriate conversation content based on the customer's reaction evaluation results.

[0769] The "means for presenting the generated talk script to the user" refers to a device or system that includes a technique for visually or audibly presenting the generated talk script to the user.

[0770] A "generative AI model" is an artificial intelligence algorithm trained using large datasets, and is a model for generating and analyzing text data.

[0771] A "prompt" is an instruction entered into a generative AI model, and is text used to control the AI's behavior and generation.

[0772] The present invention provides a system for supporting salespeople in responding to customers promptly and appropriately in meetings and business negotiations. Specific embodiments for carrying out the present invention will be described below.

[0773] This system is broadly composed of two main components: a terminal and a server.

[0774] Device configuration

[0775] The terminal is a device used in a meeting or business negotiation, and specifically includes a microphone, a camera, a voice recognition API, and an emotion analysis API.

[0776] 1. Acquire audio data:

[0777] The device uses a built-in microphone to collect audio data during the meeting, and audio capture is performed in real time.

[0778] 2. Audio data conversion:

[0779] The device converts the collected voice data into text data using the Google Speech-to-Text API or IBM Watson Speech to Text API.

[0780] 3. Emotion recognition:

[0781] The device uses a camera to capture the user's facial expressions and a microphone to capture voice tones. This data is then analyzed using emotion analysis APIs such as Microsoft Azure Emotion API and Amazon Rekognition to generate emotion data in real time.

[0782] 4. Data transmission:

[0783] The terminal buffers the generated character data and emotion data at regular intervals and transmits them to the server.

[0784] Server Configuration

[0785] The server is a system that receives data sent from the device, analyzes it, and generates a talk script, using natural language processing technology and generative AI models.

[0786] 1. Receiving data:

[0787] The server receives the character data and emotion data sent from the terminal.

[0788] 2. Data Analysis:

[0789] The server uses natural language processing technology (for example, Python's NLTK library or SpaCy) to analyze the received text data and evaluate the intent and sentiment of the customer's comments.

[0790] 3. Emotional data integration:

[0791] The server integrates the received emotional data and evaluates the customer's reaction comprehensively.

[0792] 4. Generate the talk script:

[0793] The server uses a generative AI model (e.g., OpenAI's GPT-3) based on the analysis results to generate an appropriate conversation script, which provides the optimal response method for the user.

[0794] 5. Sending the talk script:

[0795] The server transmits the generated talk script to the terminal, which then presents the talk script to the user visually or audibly.

[0796] Specific examples

[0797] Initial Setup and Starting a Meeting

[0798] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product." The device then activates its voice recognition function and emotion engine and begins recording the audio of the meeting.

[0799] Real-time transcription and analysis example

[0800] A customer says, "I think the price of this product is too high." The device transcribes this speech in real time and sends the text data to the server. The server analyzes the text data and evaluates the customer's sentiment as negative.

[0801] Example of user emotion recognition

[0802] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions, and then transmits the user's emotion data to the server.

[0803] Example of generating and presenting a talk script

[0804] The server generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance." The server sends the generated script to the terminal, which then displays it to the user.

[0805] Example of user response

[0806] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[0807] Prompt Sentence Examples

[0808] "Provide a script to alleviate the feelings of impatience identified by the emotion engine."

[0809] "How can I generate a talk script to respond to negative comments from customers?"

[0810] This system enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, enabling them to conduct effective sales activities.

[0811] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0812] Step 1: Acquire and transcribe audio data

[0813] Input: A user starts a conference and a microphone collects audio.

[0814] How it works: The device captures real-time audio during meetings, which is then sent to the Google Speech-to-Text API or IBM Watson Speech to Text API for conversion into text data.

[0815] Data processing: The audio data is converted into text data. Specifically, the API analyzes the audio waveform and generates the corresponding text.

[0816] Output: Text data (e.g., "This product is too expensive").

[0817] Step 2: Sending character data

[0818] Input: The character data generated in step 1.

[0819] Operation: The terminal buffers the generated character data at a fixed time-slide interval and transmits it to the server sequentially.

[0820] Data processing: Data is accumulated through buffering and converted into a format that can be sent in fixed amounts.

[0821] Output: Character data sent to the server.

[0822] Step 3: Parsing character data

[0823] Input: Character data sent to the server.

[0824] How it works: The server analyzes text data using Python's NLTK library, SpaCy, etc. Specifically, it performs semantic and sentiment analysis of the text.

[0825] Data processing: Grammatical analysis, keyword extraction, and sentiment analysis are performed on the text data.

[0826] Output: Analysis results (e.g., "negative sentiment", "price mentions").

[0827] Step 4: Emotion recognition and data transmission

[0828] Input: User's voice tone and facial expression data during the meeting.

[0829] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice tone, and this data is analyzed in real time using tools like the Microsoft Azure Emotion API and Amazon Rekognition.

[0830] Data processing: The captured voice tone and facial expression data are converted into emotion data through emotion analysis algorithms.

[0831] Output: Generated user emotion data (e.g., "impatience").

[0832] Step 5: Assess your emotions and reactions

[0833] Input: Analysis results of text data and emotion data.

[0834] Operation: The server integrates the results of the character data analysis with the user's emotional data to evaluate the customer's reaction. Specifically, it compares both sets of emotional data and performs a comprehensive emotional evaluation.

[0835] Data processing: By integrating text data and emotion data, more detailed emotion evaluation is performed.

[0836] Output: Customer reaction evaluation results (e.g., "Customers reacted negatively to the price").

[0837] Step 6: Generate the talk script

[0838] Input: Customer response evaluation results.

[0839] Operation: The server uses the generative AI model to generate an appropriate talk script based on the evaluation results. The server inputs the prompt sentence into the generative AI model and obtains the script.

[0840] Data processing: Generative AI models generate natural-sounding conversation scripts based on context.

[0841] Output: Generated talk script (e.g., "Price is important, but this product offers great long-term value for money").

[0842] Step 7: Send and present your talk script

[0843] Input: The generated talk script.

[0844] Operation: The server sends the generated talk script to the terminal, which receives it and presents it to the user visually or audibly.

[0845] Data processing: The talk script is converted into a format that is easy for the user to view or to use in audio.

[0846] Output: The talk script presented to the user (e.g., the script displayed on the device screen).

[0847] Through this detailed step-by-step process, the present invention helps users respond quickly and appropriately during meetings and business negotiations with customers.

[0848] (Application example 2)

[0849] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0850] It is important for sales representatives to respond quickly and appropriately to customer emotions and reactions during meetings and negotiations. However, it is difficult to understand the content of the conversation in real time, instantly grasp the customer's emotions, and provide an appropriate dialogue script that matches them. There is a need to solve this problem and significantly improve the efficiency and effectiveness of sales activities. Furthermore, to achieve real-time support, it is necessary to operate an integrated system on a device worn by the sales representative.

[0851] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0852] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for displaying the generated talk script in real time on a device worn by a sales representative. This enables the sales representative to respond appropriately on the spot during a conversation with a customer, thereby increasing the success rate of meetings and business negotiations.

[0853] "Conference audio data" refers to all audio information spoken during the conference, including conversations between the sales representative and the customer.

[0854] "Real-time transcription means" refers to a technical means for instantly converting collected audio data into text data.

[0855] "Transcription data" means audio data converted into text format.

[0856] "Means for analyzing the transcript data" refers to the technical means for analyzing the converted transcript data to understand and classify its content.

[0857] "Means for assessing emotions and reactions" refers to technological means for determining customer emotions and reactions based on analyzed transcript data.

[0858] A "talk script" is a document that provides instructions and examples of conversations that a sales representative can use to respond appropriately to a customer.

[0859] "Generative means" refers to technological means that use engines or algorithms to create new information or data for a specific purpose.

[0860] "Means for presenting" refers to a technical means for displaying or notifying the generated talk script to the sales representative.

[0861] "Means for displaying on a device in real time" refers to a technical means for instantly providing the generated talk script on the display screen of a device (such as smart glasses) worn by a sales representative.

[0862] (Embodiment of the Invention)

[0863] The present invention provides a system that enables salespeople to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system transcribes voice data in real time, analyzes the transcript to evaluate the customer's emotions and reactions, generates an appropriate talk script based on the analysis, and displays it in real time on a device worn by the salesperson.

[0864] System Configuration

[0865] 1. Terminal

[0866] The device (e.g., smart glasses) performs the following operations:

[0867] Audio data collection: The built-in microphone of the smart glasses is used to record the audio of the meeting in real time.

[0868] Transcription: Recorded audio data is transcribed in real time using the Google Cloud Speech-to-Text API.

[0869] Emotion recognition: The smart glasses' built-in camera captures the customer's facial expressions and uses the Microsoft Azure Face API to analyze their emotions in real time.

[0870] Data transmission: The buffered transcription data and emotion data are sent to the server.

[0871] 2. Server

[0872] The server performs the following operations:

[0873] Data reception: Receives transcription data and emotion data sent from the device.

[0874] Natural Language Processing: Analyze the received transcription data using natural language processing techniques (e.g., NLTK with Sentiment Analysis).

[0875] Sentiment assessment: Evaluate customer emotions and reactions based on the analysis results.

[0876] Talk script generation: Use a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[0877] Send data: Send the generated talk script to the device.

[0878] 3. User (Salesperson)

[0879] Salespeople use the system as follows:

[0880] Check the talk script: Refer to the talk script displayed on the terminal and proceed with the conversation with the customer.

[0881] Real-time response: Based on the generated conversation script, appropriate responses are provided according to the customer's emotions.

[0882] Specific examples

[0883] Initial Setup and Conference Start Example

[0884] A sales representative puts on the smart glasses and starts the system. The purpose of the meeting is set to "conclude a contract for a new product."

[0885] The smart glasses activate their voice recognition and emotion recognition engines and begin recording and capturing the meeting audio and customer facial expressions in real time.

[0886] Transcription and analysis examples

[0887] A customer says, "I think this product is priced too high."

[0888] The smart glasses transcribe this speech in real time and send the text data to a server.

[0889] The server analyzes the text data and evaluates the customer's sentiment as negative.

[0890] Examples of prompt statements

[0891] Audio Transcription: 'I think the price of this product is too high.'

[0892] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[0893] Generate the appropriate talk script.

[0894] As described above, the present invention provides a system that supports the progress of a conference in real time and supports the responses of sales representatives.

[0895] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0896] Step 1:

[0897] Audio data collection

[0898] The device (smart glasses) uses a built-in microphone to record the audio of the meeting in real time.

[0899] Input: Audio data during the meeting

[0900] Output: Recorded audio data

[0901] Specifically, the system records the conversation between the sales representative and the customer in high quality and temporarily stores it in local memory for post-processing.

[0902] Step 2:

[0903] Transcription

[0904] The device transcribes the recorded audio data in real time using the Google Cloud Speech-to-Text API.

[0905] Input: Recorded audio data

[0906] Output: Real-time transcribed text data

[0907] Specifically, audio data is sent to the API, and the resulting text data is buffered and recorded sequentially.

[0908] Step 3:

[0909] emotion recognition

[0910] The device captures the customer's facial expressions using its built-in camera, and analyzes this video data in real time using the Microsoft Azure Face API to assess the customer's emotions.

[0911] Input: Customer facial expression video data

[0912] Output: Real-time analyzed emotion data

[0913] Specifically, the device's camera captures video of the customer at regular intervals, sends the video to an API, and obtains emotion recognition results.

[0914] Step 4:

[0915] Sending data

[0916] The terminal transmits the buffered transcription data and real-time emotion data to the server at regular time-sliding intervals.

[0917] Input: Transcription text data, emotion data

[0918] Output: Send data to the server

[0919] Specifically, the buffered data is collected into packets at regular intervals and sent to a specified address on the server.

[0920] Step 5:

[0921] Natural Language Processing

[0922] The server analyzes the received transcription data using natural language processing technology (e.g., NLTK with Sentiment Analysis) to understand and classify the content of the conversation.

[0923] Input: Transcribed text data sent to the server

[0924] Output: Analysis data after natural language processing

[0925] Specifically, the text data is tokenized, and the context and sentiment of each token is analyzed to determine the meaning and sentiment of the customer's words.

[0926] Step 6:

[0927] Emotional evaluation

[0928] The server evaluates the customer's emotions and reactions based on the analysis results, which also incorporates the received emotional data.

[0929] Input: Analysis data after natural language processing, emotion data

[0930] Output: Evaluated customer sentiment and reaction data

[0931] Specifically, analytical data and emotional data are combined to determine and evaluate the actual emotions felt by the customer.

[0932] Step 7:

[0933] Generate a talk script

[0934] The server uses a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[0935] Input: Evaluated customer sentiment and reaction data

[0936] Output: Generated talk script

[0937] Specifically, the prompt sentence is input into the generative AI model, and the obtained talk script is organized and saved.

[0938] Specific prompt examples:

[0939] Audio Transcription: 'I think the price of this product is too high.'

[0940] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[0941] Generate the appropriate talk script.

[0942] Step 8:

[0943] Sending data

[0944] The server transmits the generated talk script to the terminal.

[0945] Input: Generated talk script

[0946] Output: Sending data to the terminal

[0947] Specifically, the generated talk script is sent to the specified address of the terminal.

[0948] Step 9:

[0949] Show talk script

[0950] The terminal displays the generated talk script in real time within the field of view of the salesperson.

[0951] Input: Talk script received from the server

[0952] Output: Talk script displayed to sales representatives

[0953] Specifically, the talk script will be visually displayed on the device display, allowing sales representatives to quickly refer to it.

[0954] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0955] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0956] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0957] [Third embodiment]

[0958] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0959] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0960] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0961] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0962] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0963] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0964] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0965] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0966] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0967] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0968] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0969] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0970] The present invention provides a system for enabling a sales representative to respond quickly and appropriately in meetings and business negotiations with customers. The system includes the following components.

[0971] System Configuration

[0972] 1. Terminal

[0973] The device records the audio of the meeting in real time.

[0974] The device calls a speech recognition API to transcribe the recorded audio in real time.

[0975] The device sequentially buffers the generated transcription data and sends it to the server.

[0976] 2. Server

[0977] The server receives the transcription data sent from the terminal.

[0978] The server uses natural language processing technology to analyze the transcription data.

[0979] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[0980] The server sends the generated talk script to the terminal.

[0981] 3. User (Salesperson)

[0982] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[0983] Overview of program processing

[0984] 1. Acquiring and transcribing audio data

[0985] The device collects audio data during the meeting and transcribes it in real time using a speech recognition API.

[0986] The device buffers the transcription data sequentially and transmits it to the server at regular time-sliding intervals.

[0987] 2. Analysis of the transcription data

[0988] The server receives the transcription data sent from the terminal.

[0989] The server applies natural language processing techniques to the received data and analyzes the text.

[0990] Example: Analyze the statement "I think this product is too expensive" to recognize negative evaluations of price.

[0991] 3. Evaluating emotions and reactions

[0992] The server evaluates the customer's emotions and reactions based on the analysis results.

[0993] Example: The statement "it's too expensive" is judged to indicate a negative sentiment.

[0994] 4. Generate a talk script

[0995] The server generates an appropriate talk script based on the evaluation result.

[0996] Example: Generates a script that says, "Price is certainly an important factor, but our products offer excellent long-term value for money."

[0997] 5. Send and present the talk script

[0998] The server sends the generated talk script to the terminal.

[0999] The terminal presents the talk script to the user visually or audibly.

[1000] Specific examples

[1001] Initial Setup and Starting a Meeting

[1002] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product."

[1003] The device will turn on the voice recognition function and begin recording the meeting audio.

[1004] Real-time transcription and analysis

[1005] A customer says, "I think the price of this product is too high."

[1006] The device transcribes this speech in real time and sends the text data to a server.

[1007] The server analyzes the text data and evaluates the customer's sentiment as negative.

[1008] Generate and present talk scripts

[1009] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[1010] The server sends the generated script to the terminal, which displays it to the user.

[1011] User response

[1012] Using the script provided, the user responds, "Price is certainly an important point, but our products offer excellent long-term cost performance," and continues the conversation with the customer.

[1013] In this way, the present invention provides an effective tool to assist salespeople in responding quickly and appropriately during a meeting.

[1014] The processing flow will be explained below.

[1015] Step 1:

[1016] The user starts the system and sets the goal and main agenda of the meeting. For example, the user enters "concluding a contract for a new product" as the goal.

[1017] Step 2:

[1018] The device will turn on the voice recognition function and begin recording the meeting audio in real time.

[1019] Step 3:

[1020] The device sends the recorded voice data to the speech recognition API and converts it into text data in real time. The converted text data is buffered sequentially.

[1021] Step 4:

[1022] The device sends buffered transcription data to the server at regular time-sliding intervals.

[1023] Step 5:

[1024] The server receives the transcription data sent from the device and places the received data in a queue for analysis.

[1025] Step 6:

[1026] The server uses natural language processing technology to analyze the transcription data, for example, analyzing a customer statement in the text, "I think the price of this product is too high," to recognize a negative reaction to the price.

[1027] Step 7:

[1028] The server evaluates the customer's emotions and reactions based on the analysis results. Comments such as "too expensive" are evaluated negatively, and the server determines that the enthusiasm for the meeting is declining.

[1029] Step 8:

[1030] Based on the evaluation results, the server generates an appropriate talk script, such as, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[1031] Step 9:

[1032] The server sends the generated talk script to the terminal, which is then used to continue the conference.

[1033] Step 10:

[1034] The terminal displays the generated talk script to the user, who then uses this script to speak to the customer.

[1035] Step 11:

[1036] The user responds, "Price is certainly an important factor, but our products offer excellent long-term value for money," and continues the conversation.

[1037] Step 12:

[1038] The device records the speech again and repeats the process from step 3. Each time new transcription data is generated, it is sent sequentially to the server, where the process of analysis, evaluation, and script generation continues.

[1039] In this way, the system helps manage meetings in real time, enabling salespeople to effectively serve customers.

[1040] Example 1

[1041] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1042] The lack of effective support tools that enable sales representatives to respond quickly and appropriately during meetings and negotiations with customers is an issue. Specifically, the lack of real-time feedback on customer comments and the provision of appropriate conversation scripts can reduce the success rate of sales negotiations.

[1043] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1044] In this invention, the server includes means for acquiring conference audio data in real time using a voice input device, means for transcribing in real time using a voice recognition API, means for sequentially buffering the transcribed data and transmitting it to the server at regular time slide intervals, means for receiving the transcribed data and analyzing it using natural language processing technology, means for evaluating the customer's emotions and reactions based on the analysis results, means for generating an appropriate talk script using a generative AI model based on the evaluation results, and means for presenting the generated talk script to the user. This enables sales representatives to respond quickly and appropriately to customer comments and emotions during meetings and lead sales negotiations to success.

[1045] The "voice input device" is hardware for acquiring conference voice data in real time.

[1046] A "voice recognition API" is a software interface for converting acquired voice data into text data.

[1047] "Buffering" is a process of temporarily storing transcription data and sending it to a server at regular time-slide intervals.

[1048] "Natural language processing technology" refers to algorithms and technologies for analyzing text data and understanding meaning and emotions.

[1049] A "generative AI model" is an artificial intelligence algorithm that generates an appropriate talk script based on the analysis results.

[1050] A "talk script" is a conversation script that sales representatives can use as a reference, generated based on the customer's emotions and reactions.

[1051] "Sequential" means performing a series of data or processes one step at a time in order.

[1052] The present invention provides a system that enables salespeople to respond quickly and appropriately in meetings and business negotiations with customers. This system operates in cooperation with terminals, a server, and users.

[1053] System Configuration

[1054] 1. Terminal

[1055] The terminal uses a voice input device to capture the audio data of the meeting in real time. The hardware used here is a regular smartphone or tablet.

[1056] The device uses speech recognition APIs such as the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services to transcribe in real time.

[1057] The terminal has a communication function for sequentially buffering the transcription data and transmitting it to the server at regular time-sliding intervals.

[1058] Examples:

[1059] When a user starts a meeting, they launch the application on their device and turn on the speech recognition function. Each utterance made during the meeting is transcribed in real time and continuously sent to the server.

[1060] Example prompt sentence:

[1061] "Transcribe meeting audio in real time."

[1062] 2. Server

[1063] The server receives the transcription data sent from the device. A regular web server is used here, and an RDBMS such as MySQL or PostgreSQL is used as the database.

[1064] The server analyzes the received text data using a Python natural language processing library (e.g., NLTK or SpaCy).

[1065] The server evaluates the customer's emotions and reactions based on the analysis results. For example, it determines that a comment such as "too expensive" indicates a negative evaluation of the price.

[1066] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate talk script.

[1067] Examples:

[1068] If a customer says during a meeting, "I think the price of this product is too high," the server analyzes the comment in real time, detects the negative sentiment of "high price," and then generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance."

[1069] Example prompt sentence:

[1070] "Generate a talking script to use when a customer has a negative comment about the price."

[1071] 3. User (Salesperson)

[1072] The user can proceed with the conversation by referring to the talk script provided by the device, which can either display the script visually or read it aloud.

[1073] Examples:

[1074] During the meeting, the user responds to the customer using the script displayed on the terminal, which reads, "Price is certainly an important point. However, our products offer excellent long-term cost performance."

[1075] Example prompt sentence:

[1076] "Show the generated talk script to the user."

[1077] With the above configuration, the system of the present invention enables salespeople to respond quickly and appropriately to customer comments and feelings during a meeting, leading to successful business negotiations.

[1078] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1079] Step 1:

[1080] The terminal acquires the audio data of the conference using an audio input device. The input is the audio data of the conference, and the output is the acquired audio data. Specifically, the user launches an application on the terminal and turns on the voice recognition function. The terminal records the conference audio through the microphone.

[1081] Step 2:

[1082] The device transcribes the acquired audio data in real time using a speech recognition API. The input is the audio data from step 1, and the output is the transcribed text data. This operation uses the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services. Specifically, the device sends the audio data to the API and receives the text data returned by the API.

[1083] Step 3:

[1084] The device buffers the transcription data sequentially and sends it to the server at regular time-sliding intervals. The input is the transcription data from step 2, and the output is the text data sent to the server. Specifically, the device buffers the text data obtained every 5 seconds and sends it to the server via an HTTP POST request.

[1085] Step 4:

[1086] The server receives the transcription data sent from the device. The input is the text data sent in step 3, and the output is the received transcription data. Specifically, the server receives the HTTP request and writes the data to a database (MySQL or PostgreSQL).

[1087] Step 5:

[1088] The server analyzes the received transcription data using natural language processing technology. The input is the data received in step 4, and the output is the analysis results. This operation uses Python's NLTK and SpaCy libraries. Specifically, the server tokenizes the text data and performs sentiment analysis.

[1089] Step 6:

[1090] The server evaluates the customer's emotions and reactions based on the analysis results. The input is the analysis result from step 5, and the output is the evaluation result. Specifically, the server checks keywords such as "high" and "negative" against a dictionary and sets a negative flag.

[1091] Step 7:

[1092] Based on the evaluation results, the server uses a generative AI model to generate an appropriate talk script. The input is the evaluation result from step 6, and the output is the generated talk script. This operation uses a generative AI model such as OpenAI's GPT-3. Specifically, the server inputs a prompt statement (e.g., "Please generate a talk script that can be used when making negative comments about prices") into the AI ​​model and receives the script generated by the AI ​​model.

[1093] Step 8:

[1094] The server sends the generated talk script to the terminal. The input is the talk script from step 7, and the output is the script sent to the terminal. Specifically, the server sends the talk script to the terminal using an HTTP POST request.

[1095] Step 9:

[1096] The terminal presents the received talk script to the user. The input is the talk script from step 8, and the output is the script presented to the user. Specifically, the terminal displays the script in the application's display area or provides a function to read it aloud.

[1097] (Application example 1)

[1098] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1099] In traditional sales negotiations and customer service, it was difficult for salespeople and store clerks to accurately grasp what customers were saying and how they were feeling, and then respond appropriately immediately. Furthermore, because conversations were recorded and analyzed manually, real-time support was difficult, leading to inconsistencies in the quality of customer service. This resulted in issues such as lower customer satisfaction and a lower success rate for sales negotiations.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1101] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for collecting audio data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time, thereby enabling salespeople or store clerks to receive appropriate feedback and responses in real time during conversations with customers.

[1102] "Means for transcribing meeting audio data in real time" is a technology that converts audio data collected during meetings or business negotiations into text data in real time.

[1103] "Means of analyzing transcribed data and evaluating customer emotions and reactions" refers to a technology that analyzes transcribed data using natural language processing technology and evaluates customer emotions and reactions.

[1104] "Means for generating appropriate talk scripts based on customers' emotions and reactions" refers to technology that automatically generates appropriate response content based on customers' emotions and reactions obtained through analysis.

[1105] The "means for presenting the generated talk script to the user" is a technique for immediately providing the generated response content to the user visually or audibly.

[1106] "Means for collecting voice data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time" refers to technology that collects voice data in real time during business negotiations or customer interactions, analyzes the data, and instantly generates feedback and responses based on the results, which are then provided to the user.

[1107] This invention is a system that allows salespeople and store clerks to analyze conversations with customers in real time and generate and present appropriate responses during business negotiations and customer service. This system is realized by combining technologies using speech recognition, natural language processing, and generative AI models.

[1108] Terminal

[1109] The device first collects real-time audio data during business negotiations and meetings using smart glasses or smartphones, and then transcribes the collected audio data in real time using a speech recognition API.

[1110] server

[1111] The server receives the transcription data sent from the device. The received data is analyzed using natural language processing technology to evaluate the customer's sentiment and reactions. This analysis is performed using TextBlob or a similar tool. Based on the analysis results, the server uses a generative AI model to generate an appropriate talk script. The generative AI model typically uses the OpenAI API.

[1112] Users (salespeople and store clerks)

[1113] The user receives the talk script from the device in real time and immediately applies it to the conversation with the customer. For example, if the user is using smart glasses, the talk script is displayed visually. This allows the user to provide appropriate responses without interrupting the flow of the conversation.

[1114] Specific examples

[1115] 1. Audio data collection and transcription

[1116] If a customer says, "I'd like to know more about this product" during a sales negotiation, the device (smart glasses or smartphone) will collect the voice in real time and transcribe it using a speech recognition API.

[1117] 2. Analysis of the transcription data

[1118] The device sends the transcribed text data to the server, which analyzes it using TextBlob and determines that more information is required.

[1119] 3. Generate a talk script

[1120] The server uses the generative AI model to generate a talk script that says, "This product uses the latest technology and is particularly energy-efficient."

[1121] 4. Presenting the talk script

[1122] The generated talk script is sent to the terminal and displayed on the user's (salesperson or store clerk's) smart glasses. The user can use it as a reference to provide appropriate explanations to the customer.

[1123] Prompt Sentence Examples

[1124] An example of a prompt sentence input to the generative AI model is as follows:

[1125] "Customer question: I'd like to know more about this product. Good sales associate response:"

[1126] In this way, the present invention provides an effective means for salespeople and store clerks to respond appropriately in real time during business negotiations and customer service.

[1127] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1128] Step 1:

[1129] The device collects voice data in real time during business negotiations or customer interactions. The collected voice data is then transcribed using a voice recognition API. The input for this step is the voice data from business negotiations or customer interactions, and the output is text data. This step includes the specific operation of calling the voice recognition API and converting voice to text.

[1130] Step 2:

[1131] The terminal sequentially transmits the transcribed text data to the server. The input of this step is the text data generated by speech recognition, and the output is the transcribed data transmitted to the server. This step includes specific operations of sequentially buffering and transmitting the text data to the server.

[1132] Step 3:

[1133] The server receives the transcription data sent from the terminal. The input of this step is the transcription data sent from the terminal, and the output is the received transcription data. The specific operation of the server receiving data over the network is included.

[1134] Step 4:

[1135] The server applies natural language processing techniques to the received transcription data to analyze the customer's emotions and reactions. The input of this step is the received transcription data, and the output is the analysis results (customer's emotions and reactions). Specifically, it uses TextBlob to analyze the customer's utterances and perform emotion evaluation.

[1136] Step 5:

[1137] The server generates an appropriate talk script based on the analysis results. The input to this step is the analysis results, and the output is the generated talk script. This step includes specific operations to generate an appropriate response using a generative AI model. The prompt sentence is input into the generative AI model to generate a talk script.

[1138] Step 6:

[1139] The server sends the generated talk script to the terminal. The input of this step is the generated talk script, and the output is the talk script sent to the terminal. This step includes a specific operation of sending the talk script to the terminal over the network.

[1140] Step 7:

[1141] The terminal presents the received talk script to the user (salesperson or store clerk). The input of this step is the talk script sent from the server, and the output is the talk script presented to the user. This step includes the specific operation of displaying the talk script on the screen of smart glasses or a smartphone.

[1142] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1143] The present invention provides a system that enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system includes a means for transcribing conference audio data in real time, analyzing the transcript to evaluate customer emotions and reactions, and generating an appropriate talk script based on the results, which is then presented to the user. The system also includes an emotion engine that recognizes the user's emotions, enabling the generation of a talk script that takes the user's emotions into consideration.

[1144] System Configuration

[1145] 1. Terminal

[1146] The device records the audio of the meeting in real time.

[1147] The device sends the recorded audio to a speech recognition API, which transcribes it in real time.

[1148] The device buffers the transcription data sequentially and sends it to the server.

[1149] The device uses an emotion engine to analyze the user's facial expressions and voice tone and recognize their emotions.

[1150] 2. Server

[1151] The server receives the transcription data sent from the terminal.

[1152] The server uses natural language processing techniques to analyze the transcription data.

[1153] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[1154] It also integrates emotional data obtained from the user's emotion engine and dynamically updates the talk script.

[1155] The server sends the generated talk script to the terminal.

[1156] 3. User (Salesperson)

[1157] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[1158] The user's emotions are recognized and the data is used for analysis by the emotion engine.

[1159] Overview of program processing

[1160] 1. Acquiring and transcribing audio data

[1161] The device collects voice data during the meeting and converts it into text in real time using a speech recognition API.

[1162] 2. Sending the transcription data

[1163] The device sequentially buffers the generated transcription data and transmits it to the server at regular time-sliding intervals.

[1164] 3. Analysis of Transcription Data

[1165] The server receives the transcription data sent from the device and applies natural language processing techniques to analyze the text.

[1166] 4. User Emotion Recognition by Emotion Engine

[1167] The device uses an emotion engine to analyze the user's facial expressions and voice tone during the meeting and generate the user's emotion data in real time.

[1168] The terminal transmits the user's emotion data to the server.

[1169] 5. Evaluating emotions and reactions

[1170] The server integrates the analysis results with the user's emotional data to evaluate the customer's emotions and reactions.

[1171] For example, when a customer makes a negative comment such as "This product is too expensive," the emotion engine recognizes the user's impatience and confusion.

[1172] 6. Generating a Talk Script

[1173] The server generates an appropriate talk script based on the evaluation results, for example, a script that explains to the user, "Price is an important point, but our products offer excellent long-term cost performance."

[1174] 7. Sending the talk script

[1175] The server transmits the generated talk script to the terminal.

[1176] The terminal presents the talk script to the user visually or audibly.

[1177] Specific examples

[1178] Initial Setup and Starting a Meeting

[1179] The user starts the system and sets "concluding a contract for a new product" as the goal of the meeting.

[1180] The device activates its voice recognition and emotion engine and begins recording the meeting audio.

[1181] Real-time transcription and analysis

[1182] A customer says, "I think the price of this product is too high."

[1183] The device transcribes this speech in real time and sends the text data to a server.

[1184] The server analyzes the text data and evaluates the customer's sentiment as negative.

[1185] User Emotion Recognition

[1186] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions.

[1187] The terminal transmits the user's emotion data to the server.

[1188] Generate and present talk scripts

[1189] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[1190] The server sends the generated script to the terminal, which displays it to the user.

[1191] User response

[1192] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[1193] In this way, the present invention provides a system that supports the progress of a conference in real time and helps users take appropriate action while taking into account their emotions.

[1194] The processing flow will be explained below.

[1195] Step 1:

[1196] The user starts the system and sets the goal and main agenda of the meeting. For example, the user can enter "concluding a contract for a new product" as the goal.

[1197] Step 2:

[1198] The device will turn on its voice recognition and emotion engine and begin recording the meeting audio in real time.

[1199] Step 3:

[1200] The device sends voice data to the speech recognition API in real time and transcribes it. The transcribed text data is then buffered sequentially.

[1201] Step 4:

[1202] The device sends buffered transcription data to the server at regular time-sliding intervals.

[1203] Step 5:

[1204] The server receives the transcription data sent from the device and places it in an analysis queue.

[1205] Step 6:

[1206] The server uses natural language processing technology to analyze the received transcription data, for example, analyzing a statement such as "I think the price of this product is too high" to recognize a negative reaction to the price.

[1207] Step 7:

[1208] The device uses the user's facial recognition camera and microphone to analyze the user's facial expressions and voice tone using an emotion engine, generating the user's emotion data in real time.

[1209] Step 8:

[1210] The user's emotion data generated by the terminal is transmitted to the server.

[1211] Step 9:

[1212] The server combines the analysis results of the transcription data with the user's emotional data to evaluate the customer's emotions and reactions. For example, it may evaluate a comment such as "it's too expensive" as negative and determine that the user's emotions are impatience or confusion.

[1213] Step 10:

[1214] The server generates an appropriate talk script based on the evaluation results, for example, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[1215] Step 11:

[1216] The server sends the generated talk script to the terminal.

[1217] Step 12:

[1218] The terminal presents the generated talk script to the user visually or audibly.

[1219] Step 13:

[1220] The user refers to the provided talk script and replies, "Price is important, but this product offers excellent long-term cost performance," and continues the conversation.

[1221] Step 14:

[1222] The device records the audio again and repeats the process from step 3 onwards, generating new transcripts sequentially and sending them to the server, where the process of analysis, evaluation and script generation continues.

[1223] In this way, this system supports the progress of meetings in real time and enables appropriate responses to be made taking into account the feelings of both the user and the customer.

[1224] Example 2

[1225] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1226] Conventional meeting support systems were unable to transcribe voice data in real time or perform subsequent analysis quickly and accurately. Furthermore, it was difficult to generate appropriate talk scripts that took into account the customer's emotions and reactions, placing a heavy burden on sales representatives. Furthermore, there were insufficient means to dynamically respond to the progress of the meeting and the user's emotions, making it difficult to conduct effective sales activities.

[1227] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring conference audio in real time and converting it into text data, means for transmitting the text data to the server at regular intervals, means for analyzing the text data using natural language processing technology, means for analyzing the emotions of the user and the customer in real time using an emotion analysis engine, means for integrating the emotion data of the customer and the user and evaluating the customer's reaction, means for generating a talk script based on the evaluation result, and means for presenting the generated talk script to the user. This enables real-time analysis of audio data and rapid generation of a talk script according to the customer's emotion.

[1228] "Means for capturing conference audio in real time and converting it into text data" refers to a device or system that includes technology for instantly capturing audio generated during a conference as digital data and converting that audio data into text data.

[1229] The "means for transmitting character data to a server at regular intervals" refers to a device or system for transmitting the generated character data to a server at appropriate timing.

[1230] "Means for analyzing text data using natural language processing techniques" refers to a device or system that uses natural language processing algorithms and techniques to perform text analysis on acquired text data.

[1231] "Means for analyzing user and customer emotions in real time using an emotion analysis engine" refers to a device or system that includes technology for analyzing the voice tone and facial expressions of users and customers and evaluating their emotions in real time.

[1232] "Means for integrating emotional data of customers and users and evaluating customer reactions" refers to a device or system that includes technology for integrating emotional data obtained from users and customers and evaluating customer reactions based on the results.

[1233] The "means for generating a talk script based on the evaluation results" refers to a device or system including a technology for generating a talk script including appropriate conversation content based on the customer's reaction evaluation results.

[1234] The "means for presenting the generated talk script to the user" refers to a device or system that includes a technique for visually or audibly presenting the generated talk script to the user.

[1235] A "generative AI model" is an artificial intelligence algorithm trained using large datasets, and is a model for generating and analyzing text data.

[1236] A "prompt" is an instruction entered into a generative AI model, and is text used to control the AI's behavior and generation.

[1237] The present invention provides a system for supporting salespeople in responding to customers promptly and appropriately in meetings and business negotiations. Specific embodiments for carrying out the present invention will be described below.

[1238] This system is broadly composed of two main components: a terminal and a server.

[1239] Device configuration

[1240] The terminal is a device used in a meeting or business negotiation, and specifically includes a microphone, a camera, a voice recognition API, and an emotion analysis API.

[1241] 1. Acquire audio data:

[1242] The device uses a built-in microphone to collect audio data during the meeting, and audio capture is performed in real time.

[1243] 2. Audio data conversion:

[1244] The device converts the collected voice data into text data using the Google Speech-to-Text API or IBM Watson Speech to Text API.

[1245] 3. Emotion recognition:

[1246] The device uses a camera to capture the user's facial expressions and a microphone to capture voice tones. This data is then analyzed using emotion analysis APIs such as Microsoft Azure Emotion API and Amazon Rekognition to generate emotion data in real time.

[1247] 4. Data transmission:

[1248] The terminal buffers the generated character data and emotion data at regular intervals and transmits them to the server.

[1249] Server Configuration

[1250] The server is a system that receives data sent from the device, analyzes it, and generates a talk script, using natural language processing technology and generative AI models.

[1251] 1. Receiving data:

[1252] The server receives the character data and emotion data sent from the terminal.

[1253] 2. Data Analysis:

[1254] The server uses natural language processing technology (for example, Python's NLTK library or SpaCy) to analyze the received text data and evaluate the intent and sentiment of the customer's comments.

[1255] 3. Emotional data integration:

[1256] The server integrates the received emotional data and evaluates the customer's reaction comprehensively.

[1257] 4. Generate the talk script:

[1258] The server uses a generative AI model (e.g., OpenAI's GPT-3) based on the analysis results to generate an appropriate conversation script, which provides the optimal response method for the user.

[1259] 5. Sending the talk script:

[1260] The server transmits the generated talk script to the terminal, which then presents the talk script to the user visually or audibly.

[1261] Specific examples

[1262] Initial Setup and Starting a Meeting

[1263] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product." The device then activates its voice recognition function and emotion engine and begins recording the audio of the meeting.

[1264] Real-time transcription and analysis example

[1265] A customer says, "I think the price of this product is too high." The device transcribes this speech in real time and sends the text data to the server. The server analyzes the text data and evaluates the customer's sentiment as negative.

[1266] Example of user emotion recognition

[1267] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions, and then transmits the user's emotion data to the server.

[1268] Example of generating and presenting a talk script

[1269] The server generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance." The server sends the generated script to the terminal, which then displays it to the user.

[1270] Example of user response

[1271] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[1272] Prompt Sentence Examples

[1273] "Provide a script to alleviate the feelings of impatience identified by the emotion engine."

[1274] "How can I generate a talk script to respond to negative comments from customers?"

[1275] This system enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, enabling them to conduct effective sales activities.

[1276] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1277] Step 1: Acquire and transcribe audio data

[1278] Input: A user starts a conference and a microphone collects audio.

[1279] How it works: The device captures real-time audio during meetings, which is then sent to the Google Speech-to-Text API or IBM Watson Speech to Text API for conversion into text data.

[1280] Data processing: The audio data is converted into text data. Specifically, the API analyzes the audio waveform and generates the corresponding text.

[1281] Output: Text data (e.g., "This product is too expensive").

[1282] Step 2: Sending character data

[1283] Input: The character data generated in step 1.

[1284] Operation: The terminal buffers the generated character data at a fixed time-slide interval and transmits it to the server sequentially.

[1285] Data processing: Data is accumulated through buffering and converted into a format that can be sent in fixed amounts.

[1286] Output: Character data sent to the server.

[1287] Step 3: Parsing character data

[1288] Input: Character data sent to the server.

[1289] How it works: The server analyzes text data using Python's NLTK library, SpaCy, etc. Specifically, it performs semantic and sentiment analysis of the text.

[1290] Data processing: Grammatical analysis, keyword extraction, and sentiment analysis are performed on the text data.

[1291] Output: Analysis results (e.g., "negative sentiment", "price mentions").

[1292] Step 4: Emotion recognition and data transmission

[1293] Input: User's voice tone and facial expression data during the meeting.

[1294] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice tone, and this data is analyzed in real time using tools like the Microsoft Azure Emotion API and Amazon Rekognition.

[1295] Data processing: The captured voice tone and facial expression data are converted into emotion data through emotion analysis algorithms.

[1296] Output: Generated user emotion data (e.g., "impatience").

[1297] Step 5: Assess your emotions and reactions

[1298] Input: Analysis results of text data and emotion data.

[1299] Operation: The server integrates the results of the character data analysis with the user's emotional data to evaluate the customer's reaction. Specifically, it compares both sets of emotional data and performs a comprehensive emotional evaluation.

[1300] Data processing: By integrating text data and emotion data, more detailed emotion evaluation is performed.

[1301] Output: Customer reaction evaluation results (e.g., "Customers reacted negatively to the price").

[1302] Step 6: Generate the talk script

[1303] Input: Customer response evaluation results.

[1304] Operation: The server uses the generative AI model to generate an appropriate talk script based on the evaluation results. The server inputs the prompt sentence into the generative AI model and obtains the script.

[1305] Data processing: Generative AI models generate natural-sounding conversation scripts based on context.

[1306] Output: Generated talk script (e.g., "Price is important, but this product offers great long-term value for money").

[1307] Step 7: Send and present your talk script

[1308] Input: The generated talk script.

[1309] Operation: The server sends the generated talk script to the terminal, which receives it and presents it to the user visually or audibly.

[1310] Data processing: The talk script is converted into a format that is easy for the user to view or to use in audio.

[1311] Output: The talk script presented to the user (e.g., the script displayed on the device screen).

[1312] Through this detailed step-by-step process, the present invention helps users respond quickly and appropriately during meetings and business negotiations with customers.

[1313] (Application example 2)

[1314] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1315] It is important for sales representatives to respond quickly and appropriately to customer emotions and reactions during meetings and negotiations. However, it is difficult to understand the content of the conversation in real time, instantly grasp the customer's emotions, and provide an appropriate dialogue script that matches them. There is a need to solve this problem and significantly improve the efficiency and effectiveness of sales activities. Furthermore, to achieve real-time support, it is necessary to operate an integrated system on a device worn by the sales representative.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1317] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for displaying the generated talk script in real time on a device worn by a sales representative. This enables the sales representative to respond appropriately on the spot during a conversation with a customer, thereby increasing the success rate of meetings and business negotiations.

[1318] "Conference audio data" refers to all audio information spoken during the conference, including conversations between the sales representative and the customer.

[1319] "Real-time transcription means" refers to a technical means for instantly converting collected audio data into text data.

[1320] "Transcription data" means audio data converted into text format.

[1321] "Means for analyzing the transcript data" refers to the technical means for analyzing the converted transcript data to understand and classify its content.

[1322] "Means for assessing emotions and reactions" refers to technological means for determining customer emotions and reactions based on analyzed transcript data.

[1323] A "talk script" is a document that provides instructions and examples of conversations that a sales representative can use to respond appropriately to a customer.

[1324] "Generative means" refers to technological means that use engines or algorithms to create new information or data for a specific purpose.

[1325] "Means for presenting" refers to a technical means for displaying or notifying the generated talk script to the sales representative.

[1326] "Means for displaying on a device in real time" refers to a technical means for instantly providing the generated talk script on the display screen of a device (such as smart glasses) worn by a sales representative.

[1327] (Embodiment of the Invention)

[1328] The present invention provides a system that enables salespeople to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system transcribes voice data in real time, analyzes the transcript to evaluate the customer's emotions and reactions, generates an appropriate talk script based on the analysis, and displays it in real time on a device worn by the salesperson.

[1329] System Configuration

[1330] 1. Terminal

[1331] The device (e.g., smart glasses) performs the following operations:

[1332] Audio data collection: The built-in microphone of the smart glasses is used to record the audio of the meeting in real time.

[1333] Transcription: Recorded audio data is transcribed in real time using the Google Cloud Speech-to-Text API.

[1334] Emotion recognition: The smart glasses' built-in camera captures the customer's facial expressions and uses the Microsoft Azure Face API to analyze their emotions in real time.

[1335] Data transmission: The buffered transcription data and emotion data are sent to the server.

[1336] 2. Server

[1337] The server performs the following operations:

[1338] Data reception: Receives transcription data and emotion data sent from the device.

[1339] Natural Language Processing: Analyze the received transcription data using natural language processing techniques (e.g., NLTK with Sentiment Analysis).

[1340] Sentiment assessment: Evaluate customer emotions and reactions based on the analysis results.

[1341] Talk script generation: Use a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[1342] Send data: Send the generated talk script to the device.

[1343] 3. User (Salesperson)

[1344] Salespeople use the system as follows:

[1345] Check the talk script: Refer to the talk script displayed on the terminal and proceed with the conversation with the customer.

[1346] Real-time response: Based on the generated conversation script, appropriate responses are provided according to the customer's emotions.

[1347] Specific examples

[1348] Initial Setup and Conference Start Example

[1349] A sales representative puts on the smart glasses and starts the system. The purpose of the meeting is set to "conclude a contract for a new product."

[1350] The smart glasses activate their voice recognition and emotion recognition engines and begin recording and capturing the meeting audio and customer facial expressions in real time.

[1351] Transcription and analysis examples

[1352] A customer says, "I think this product is priced too high."

[1353] The smart glasses transcribe this speech in real time and send the text data to a server.

[1354] The server analyzes the text data and evaluates the customer's sentiment as negative.

[1355] Examples of prompt statements

[1356] Audio Transcription: 'I think the price of this product is too high.'

[1357] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[1358] Generate the appropriate talk script.

[1359] As described above, the present invention provides a system that supports the progress of a conference in real time and supports the responses of sales representatives.

[1360] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1361] Step 1:

[1362] Audio data collection

[1363] The device (smart glasses) uses a built-in microphone to record the audio of the meeting in real time.

[1364] Input: Audio data during the meeting

[1365] Output: Recorded audio data

[1366] Specifically, the system records the conversation between the sales representative and the customer in high quality and temporarily stores it in local memory for post-processing.

[1367] Step 2:

[1368] Transcription

[1369] The device transcribes the recorded audio data in real time using the Google Cloud Speech-to-Text API.

[1370] Input: Recorded audio data

[1371] Output: Real-time transcribed text data

[1372] Specifically, audio data is sent to the API, and the resulting text data is buffered and recorded sequentially.

[1373] Step 3:

[1374] emotion recognition

[1375] The device captures the customer's facial expressions using its built-in camera, and analyzes this video data in real time using the Microsoft Azure Face API to assess the customer's emotions.

[1376] Input: Customer facial expression video data

[1377] Output: Real-time analyzed emotion data

[1378] Specifically, the device's camera captures video of the customer at regular intervals, sends the video to an API, and obtains emotion recognition results.

[1379] Step 4:

[1380] Sending data

[1381] The terminal transmits the buffered transcription data and real-time emotion data to the server at regular time-sliding intervals.

[1382] Input: Transcription text data, emotion data

[1383] Output: Send data to the server

[1384] Specifically, the buffered data is collected into packets at regular intervals and sent to a specified address on the server.

[1385] Step 5:

[1386] Natural Language Processing

[1387] The server analyzes the received transcription data using natural language processing technology (e.g., NLTK with Sentiment Analysis) to understand and classify the content of the conversation.

[1388] Input: Transcribed text data sent to the server

[1389] Output: Analysis data after natural language processing

[1390] Specifically, the text data is tokenized, and the context and sentiment of each token is analyzed to determine the meaning and sentiment of the customer's words.

[1391] Step 6:

[1392] Emotional evaluation

[1393] The server evaluates the customer's emotions and reactions based on the analysis results, which also incorporates the received emotional data.

[1394] Input: Analysis data after natural language processing, emotion data

[1395] Output: Evaluated customer sentiment and reaction data

[1396] Specifically, analytical data and emotional data are combined to determine and evaluate the actual emotions felt by the customer.

[1397] Step 7:

[1398] Generate a talk script

[1399] The server uses a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[1400] Input: Evaluated customer sentiment and reaction data

[1401] Output: Generated talk script

[1402] Specifically, the prompt sentence is input into the generative AI model, and the obtained talk script is organized and saved.

[1403] Specific prompt examples:

[1404] Audio Transcription: 'I think the price of this product is too high.'

[1405] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[1406] Generate the appropriate talk script.

[1407] Step 8:

[1408] Sending data

[1409] The server transmits the generated talk script to the terminal.

[1410] Input: Generated talk script

[1411] Output: Sending data to the terminal

[1412] Specifically, the generated talk script is sent to the specified address of the terminal.

[1413] Step 9:

[1414] Show talk script

[1415] The terminal displays the generated talk script in real time within the field of view of the salesperson.

[1416] Input: Talk script received from the server

[1417] Output: Talk script displayed to sales representatives

[1418] Specifically, the talk script will be visually displayed on the device display, allowing sales representatives to quickly refer to it.

[1419] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1421] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1422] [Fourth embodiment]

[1423] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1424] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1426] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1430] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1431] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1432] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1433] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1434] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1435] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1436] The present invention provides a system for enabling a sales representative to respond quickly and appropriately in meetings and business negotiations with customers. The system includes the following components.

[1437] System Configuration

[1438] 1. Terminal

[1439] The device records the audio of the meeting in real time.

[1440] The device calls a speech recognition API to transcribe the recorded audio in real time.

[1441] The device sequentially buffers the generated transcription data and sends it to the server.

[1442] 2. Server

[1443] The server receives the transcription data sent from the terminal.

[1444] The server uses natural language processing technology to analyze the transcription data.

[1445] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[1446] The server sends the generated talk script to the terminal.

[1447] 3. User (Salesperson)

[1448] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[1449] Overview of program processing

[1450] 1. Acquiring and transcribing audio data

[1451] The device collects audio data during the meeting and transcribes it in real time using a speech recognition API.

[1452] The device buffers the transcription data sequentially and transmits it to the server at regular time-sliding intervals.

[1453] 2. Analysis of the transcription data

[1454] The server receives the transcription data sent from the terminal.

[1455] The server applies natural language processing techniques to the received data and analyzes the text.

[1456] Example: Analyze the statement "I think this product is too expensive" to recognize negative evaluations of price.

[1457] 3. Evaluating emotions and reactions

[1458] The server evaluates the customer's emotions and reactions based on the analysis results.

[1459] Example: The statement "it's too expensive" is judged to indicate a negative sentiment.

[1460] 4. Generate a talk script

[1461] The server generates an appropriate talk script based on the evaluation result.

[1462] Example: Generates a script that says, "Price is certainly an important factor, but our products offer excellent long-term value for money."

[1463] 5. Send and present the talk script

[1464] The server sends the generated talk script to the terminal.

[1465] The terminal presents the talk script to the user visually or audibly.

[1466] Specific examples

[1467] Initial Setup and Starting a Meeting

[1468] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product."

[1469] The device will turn on the voice recognition function and begin recording the meeting audio.

[1470] Real-time transcription and analysis

[1471] A customer says, "I think the price of this product is too high."

[1472] The device transcribes this speech in real time and sends the text data to a server.

[1473] The server analyzes the text data and evaluates the customer's sentiment as negative.

[1474] Generate and present talk scripts

[1475] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[1476] The server sends the generated script to the terminal, which displays it to the user.

[1477] User response

[1478] Using the script provided, the user responds, "Price is certainly an important point, but our products offer excellent long-term cost performance," and continues the conversation with the customer.

[1479] In this way, the present invention provides an effective tool to assist salespeople in responding quickly and appropriately during a meeting.

[1480] The processing flow will be explained below.

[1481] Step 1:

[1482] The user starts the system and sets the goal and main agenda of the meeting. For example, the user enters "concluding a contract for a new product" as the goal.

[1483] Step 2:

[1484] The device will turn on the voice recognition function and begin recording the meeting audio in real time.

[1485] Step 3:

[1486] The device sends the recorded voice data to the speech recognition API and converts it into text data in real time. The converted text data is buffered sequentially.

[1487] Step 4:

[1488] The device sends buffered transcription data to the server at regular time-sliding intervals.

[1489] Step 5:

[1490] The server receives the transcription data sent from the device and places the received data in a queue for analysis.

[1491] Step 6:

[1492] The server uses natural language processing technology to analyze the transcription data, for example, analyzing a customer statement in the text, "I think the price of this product is too high," to recognize a negative reaction to the price.

[1493] Step 7:

[1494] The server evaluates the customer's emotions and reactions based on the analysis results. Comments such as "too expensive" are evaluated negatively, and the server determines that the enthusiasm for the meeting is declining.

[1495] Step 8:

[1496] Based on the evaluation results, the server generates an appropriate talk script, such as, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[1497] Step 9:

[1498] The server sends the generated talk script to the terminal, which is then used to continue the conference.

[1499] Step 10:

[1500] The terminal displays the generated talk script to the user, who then uses this script to speak to the customer.

[1501] Step 11:

[1502] The user responds, "Price is certainly an important factor, but our products offer excellent long-term value for money," and continues the conversation.

[1503] Step 12:

[1504] The device records the speech again and repeats the process from step 3. Each time new transcription data is generated, it is sent sequentially to the server, where the process of analysis, evaluation, and script generation continues.

[1505] In this way, the system helps manage meetings in real time, enabling salespeople to effectively serve customers.

[1506] Example 1

[1507] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1508] The lack of effective support tools that enable sales representatives to respond quickly and appropriately during meetings and negotiations with customers is an issue. Specifically, the lack of real-time feedback on customer comments and the provision of appropriate conversation scripts can reduce the success rate of sales negotiations.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1510] In this invention, the server includes means for acquiring conference audio data in real time using a voice input device, means for transcribing in real time using a voice recognition API, means for sequentially buffering the transcribed data and transmitting it to the server at regular time slide intervals, means for receiving the transcribed data and analyzing it using natural language processing technology, means for evaluating the customer's emotions and reactions based on the analysis results, means for generating an appropriate talk script using a generative AI model based on the evaluation results, and means for presenting the generated talk script to the user. This enables sales representatives to respond quickly and appropriately to customer comments and emotions during meetings and lead sales negotiations to success.

[1511] The "voice input device" is hardware for acquiring conference voice data in real time.

[1512] A "voice recognition API" is a software interface for converting acquired voice data into text data.

[1513] "Buffering" is a process of temporarily storing transcription data and sending it to a server at regular time-slide intervals.

[1514] "Natural language processing technology" refers to algorithms and technologies for analyzing text data and understanding meaning and emotions.

[1515] A "generative AI model" is an artificial intelligence algorithm that generates an appropriate talk script based on the analysis results.

[1516] A "talk script" is a conversation script that sales representatives can use as a reference, generated based on the customer's emotions and reactions.

[1517] "Sequential" means performing a series of data or processes one step at a time in order.

[1518] The present invention provides a system that enables salespeople to respond quickly and appropriately in meetings and business negotiations with customers. This system operates in cooperation with terminals, a server, and users.

[1519] System Configuration

[1520] 1. Terminal

[1521] The terminal uses a voice input device to capture the audio data of the meeting in real time. The hardware used here is a regular smartphone or tablet.

[1522] The device uses speech recognition APIs such as the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services to transcribe in real time.

[1523] The terminal has a communication function for sequentially buffering the transcription data and transmitting it to the server at regular time-sliding intervals.

[1524] Examples:

[1525] When a user starts a meeting, they launch the application on their device and turn on the speech recognition function. Each utterance made during the meeting is transcribed in real time and continuously sent to the server.

[1526] Example prompt sentence:

[1527] "Transcribe meeting audio in real time."

[1528] 2. Server

[1529] The server receives the transcription data sent from the device. A regular web server is used here, and an RDBMS such as MySQL or PostgreSQL is used as the database.

[1530] The server analyzes the received text data using a Python natural language processing library (e.g., NLTK or SpaCy).

[1531] The server evaluates the customer's emotions and reactions based on the analysis results. For example, it determines that a comment such as "too expensive" indicates a negative evaluation of the price.

[1532] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate talk script.

[1533] Examples:

[1534] If a customer says during a meeting, "I think the price of this product is too high," the server analyzes the comment in real time, detects the negative sentiment of "high price," and then generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance."

[1535] Example prompt sentence:

[1536] "Generate a talking script to use when a customer has a negative comment about the price."

[1537] 3. User (Salesperson)

[1538] The user can proceed with the conversation by referring to the talk script provided by the device, which can either display the script visually or read it aloud.

[1539] Examples:

[1540] During the meeting, the user responds to the customer using the script displayed on the terminal, which reads, "Price is certainly an important point. However, our products offer excellent long-term cost performance."

[1541] Example prompt sentence:

[1542] "Show the generated talk script to the user."

[1543] With the above configuration, the system of the present invention enables salespeople to respond quickly and appropriately to customer comments and feelings during a meeting, leading to successful business negotiations.

[1544] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1545] Step 1:

[1546] The terminal acquires the audio data of the conference using an audio input device. The input is the audio data of the conference, and the output is the acquired audio data. Specifically, the user launches an application on the terminal and turns on the voice recognition function. The terminal records the conference audio through the microphone.

[1547] Step 2:

[1548] The device transcribes the acquired audio data in real time using a speech recognition API. The input is the audio data from step 1, and the output is the transcribed text data. This operation uses the Google Cloud Speech-to-Text API and Microsoft Azure Cognitive Services. Specifically, the device sends the audio data to the API and receives the text data returned by the API.

[1549] Step 3:

[1550] The device buffers the transcription data sequentially and sends it to the server at regular time-sliding intervals. The input is the transcription data from step 2, and the output is the text data sent to the server. Specifically, the device buffers the text data obtained every 5 seconds and sends it to the server via an HTTP POST request.

[1551] Step 4:

[1552] The server receives the transcription data sent from the device. The input is the text data sent in step 3, and the output is the received transcription data. Specifically, the server receives the HTTP request and writes the data to a database (MySQL or PostgreSQL).

[1553] Step 5:

[1554] The server analyzes the received transcription data using natural language processing technology. The input is the data received in step 4, and the output is the analysis results. This operation uses Python's NLTK and SpaCy libraries. Specifically, the server tokenizes the text data and performs sentiment analysis.

[1555] Step 6:

[1556] The server evaluates the customer's emotions and reactions based on the analysis results. The input is the analysis result from step 5, and the output is the evaluation result. Specifically, the server checks keywords such as "high" and "negative" against a dictionary and sets a negative flag.

[1557] Step 7:

[1558] Based on the evaluation results, the server uses a generative AI model to generate an appropriate talk script. The input is the evaluation result from step 6, and the output is the generated talk script. This operation uses a generative AI model such as OpenAI's GPT-3. Specifically, the server inputs a prompt statement (e.g., "Please generate a talk script that can be used when making negative comments about prices") into the AI ​​model and receives the script generated by the AI ​​model.

[1559] Step 8:

[1560] The server sends the generated talk script to the terminal. The input is the talk script from step 7, and the output is the script sent to the terminal. Specifically, the server sends the talk script to the terminal using an HTTP POST request.

[1561] Step 9:

[1562] The terminal presents the received talk script to the user. The input is the talk script from step 8, and the output is the script presented to the user. Specifically, the terminal displays the script in the application's display area or provides a function to read it aloud.

[1563] (Application example 1)

[1564] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1565] In traditional sales negotiations and customer service, it was difficult for salespeople and store clerks to accurately grasp what customers were saying and how they were feeling, and then respond appropriately immediately. Furthermore, because conversations were recorded and analyzed manually, real-time support was difficult, leading to inconsistencies in the quality of customer service. This resulted in issues such as lower customer satisfaction and a lower success rate for sales negotiations.

[1566] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1567] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for collecting audio data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time, thereby enabling salespeople or store clerks to receive appropriate feedback and responses in real time during conversations with customers.

[1568] "Means for transcribing meeting audio data in real time" is a technology that converts audio data collected during meetings or business negotiations into text data in real time.

[1569] "Means of analyzing transcribed data and evaluating customer emotions and reactions" refers to a technology that analyzes transcribed data using natural language processing technology and evaluates customer emotions and reactions.

[1570] "Means for generating appropriate talk scripts based on customers' emotions and reactions" refers to technology that automatically generates appropriate response content based on customers' emotions and reactions obtained through analysis.

[1571] The "means for presenting the generated talk script to the user" is a technique for immediately providing the generated response content to the user visually or audibly.

[1572] "Means for collecting voice data during business negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time" refers to technology that collects voice data in real time during business negotiations or customer interactions, analyzes the data, and instantly generates feedback and responses based on the results, which are then provided to the user.

[1573] This invention is a system that allows salespeople and store clerks to analyze conversations with customers in real time and generate and present appropriate responses during business negotiations and customer service. This system is realized by combining technologies using speech recognition, natural language processing, and generative AI models.

[1574] Terminal

[1575] The device first collects real-time audio data during business negotiations and meetings using smart glasses or smartphones, and then transcribes the collected audio data in real time using a speech recognition API.

[1576] server

[1577] The server receives the transcription data sent from the device. The received data is analyzed using natural language processing technology to evaluate the customer's sentiment and reactions. This analysis is performed using TextBlob or a similar tool. Based on the analysis results, the server uses a generative AI model to generate an appropriate talk script. The generative AI model typically uses the OpenAI API.

[1578] Users (salespeople and store clerks)

[1579] The user receives the talk script from the device in real time and immediately applies it to the conversation with the customer. For example, if the user is using smart glasses, the talk script is displayed visually. This allows the user to provide appropriate responses without interrupting the flow of the conversation.

[1580] Specific examples

[1581] 1. Audio data collection and transcription

[1582] If a customer says, "I'd like to know more about this product" during a sales negotiation, the device (smart glasses or smartphone) will collect the voice in real time and transcribe it using a speech recognition API.

[1583] 2. Analysis of the transcription data

[1584] The device sends the transcribed text data to the server, which analyzes it using TextBlob and determines that more information is required.

[1585] 3. Generate a talk script

[1586] The server uses the generative AI model to generate a talk script that says, "This product uses the latest technology and is particularly energy-efficient."

[1587] 4. Presenting the talk script

[1588] The generated talk script is sent to the terminal and displayed on the user's (salesperson or store clerk's) smart glasses. The user can use it as a reference to provide appropriate explanations to the customer.

[1589] Prompt Sentence Examples

[1590] An example of a prompt sentence input to the generative AI model is as follows:

[1591] "Customer question: I'd like to know more about this product. Good sales associate response:"

[1592] In this way, the present invention provides an effective means for salespeople and store clerks to respond appropriately in real time during business negotiations and customer service.

[1593] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1594] Step 1:

[1595] The device collects voice data in real time during business negotiations or customer interactions. The collected voice data is then transcribed using a voice recognition API. The input for this step is the voice data from business negotiations or customer interactions, and the output is text data. This step includes the specific operation of calling the voice recognition API and converting voice to text.

[1596] Step 2:

[1597] The terminal sequentially transmits the transcribed text data to the server. The input of this step is the text data generated by speech recognition, and the output is the transcribed data transmitted to the server. This step includes specific operations of sequentially buffering and transmitting the text data to the server.

[1598] Step 3:

[1599] The server receives the transcription data sent from the terminal. The input of this step is the transcription data sent from the terminal, and the output is the received transcription data. The specific operation of the server receiving data over the network is included.

[1600] Step 4:

[1601] The server applies natural language processing techniques to the received transcription data to analyze the customer's emotions and reactions. The input of this step is the received transcription data, and the output is the analysis results (customer's emotions and reactions). Specifically, it uses TextBlob to analyze the customer's utterances and perform emotion evaluation.

[1602] Step 5:

[1603] The server generates an appropriate talk script based on the analysis results. The input to this step is the analysis results, and the output is the generated talk script. This step includes specific operations to generate an appropriate response using a generative AI model. The prompt sentence is input into the generative AI model to generate a talk script.

[1604] Step 6:

[1605] The server sends the generated talk script to the terminal. The input of this step is the generated talk script, and the output is the talk script sent to the terminal. This step includes a specific operation of sending the talk script to the terminal over the network.

[1606] Step 7:

[1607] The terminal presents the received talk script to the user (salesperson or store clerk). The input of this step is the talk script sent from the server, and the output is the talk script presented to the user. This step includes the specific operation of displaying the talk script on the screen of smart glasses or a smartphone.

[1608] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1609] The present invention provides a system that enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system includes a means for transcribing conference audio data in real time, analyzing the transcript to evaluate customer emotions and reactions, and generating an appropriate talk script based on the results, which is then presented to the user. The system also includes an emotion engine that recognizes the user's emotions, enabling the generation of a talk script that takes the user's emotions into consideration.

[1610] System Configuration

[1611] 1. Terminal

[1612] The device records the audio of the meeting in real time.

[1613] The device sends the recorded audio to a speech recognition API, which transcribes it in real time.

[1614] The device buffers the transcription data sequentially and sends it to the server.

[1615] The device uses an emotion engine to analyze the user's facial expressions and voice tone and recognize their emotions.

[1616] 2. Server

[1617] The server receives the transcription data sent from the terminal.

[1618] The server uses natural language processing techniques to analyze the transcription data.

[1619] The server evaluates the customer's emotions and reactions and generates an appropriate talk script.

[1620] It also integrates emotional data obtained from the user's emotion engine and dynamically updates the talk script.

[1621] The server sends the generated talk script to the terminal.

[1622] 3. User (Salesperson)

[1623] The user can carry out appropriate conversations by referring to the talk script presented by the terminal.

[1624] The user's emotions are recognized and the data is used for analysis by the emotion engine.

[1625] Overview of program processing

[1626] 1. Acquiring and transcribing audio data

[1627] The device collects voice data during the meeting and converts it into text in real time using a speech recognition API.

[1628] 2. Sending the transcription data

[1629] The device sequentially buffers the generated transcription data and transmits it to the server at regular time-sliding intervals.

[1630] 3. Analysis of Transcription Data

[1631] The server receives the transcription data sent from the device and applies natural language processing techniques to analyze the text.

[1632] 4. User Emotion Recognition by Emotion Engine

[1633] The device uses an emotion engine to analyze the user's facial expressions and voice tone during the meeting and generate the user's emotion data in real time.

[1634] The terminal transmits the user's emotion data to the server.

[1635] 5. Evaluating emotions and reactions

[1636] The server integrates the analysis results with the user's emotional data to evaluate the customer's emotions and reactions.

[1637] For example, when a customer makes a negative comment such as "This product is too expensive," the emotion engine recognizes the user's impatience and confusion.

[1638] 6. Generating a Talk Script

[1639] The server generates an appropriate talk script based on the evaluation results, for example, a script that explains to the user, "Price is an important point, but our products offer excellent long-term cost performance."

[1640] 7. Sending the talk script

[1641] The server transmits the generated talk script to the terminal.

[1642] The terminal presents the talk script to the user visually or audibly.

[1643] Specific examples

[1644] Initial Setup and Starting a Meeting

[1645] The user starts the system and sets "concluding a contract for a new product" as the goal of the meeting.

[1646] The device activates its voice recognition and emotion engine and begins recording the meeting audio.

[1647] Real-time transcription and analysis

[1648] A customer says, "I think the price of this product is too high."

[1649] The device transcribes this speech in real time and sends the text data to a server.

[1650] The server analyzes the text data and evaluates the customer's sentiment as negative.

[1651] User Emotion Recognition

[1652] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions.

[1653] The terminal transmits the user's emotion data to the server.

[1654] Generate and present talk scripts

[1655] The server generates a script that says, "Price is certainly an important factor, but our product offers excellent long-term value for money."

[1656] The server sends the generated script to the terminal, which displays it to the user.

[1657] User response

[1658] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[1659] In this way, the present invention provides a system that supports the progress of a conference in real time and helps users take appropriate action while taking into account their emotions.

[1660] The processing flow will be explained below.

[1661] Step 1:

[1662] The user starts the system and sets the goal and main agenda of the meeting. For example, the user can enter "concluding a contract for a new product" as the goal.

[1663] Step 2:

[1664] The device will turn on its voice recognition and emotion engine and begin recording the meeting audio in real time.

[1665] Step 3:

[1666] The device sends voice data to the speech recognition API in real time and transcribes it. The transcribed text data is then buffered sequentially.

[1667] Step 4:

[1668] The device sends buffered transcription data to the server at regular time-sliding intervals.

[1669] Step 5:

[1670] The server receives the transcription data sent from the device and places it in an analysis queue.

[1671] Step 6:

[1672] The server uses natural language processing technology to analyze the received transcription data, for example, analyzing a statement such as "I think the price of this product is too high" to recognize a negative reaction to the price.

[1673] Step 7:

[1674] The device uses the user's facial recognition camera and microphone to analyze the user's facial expressions and voice tone using an emotion engine, generating the user's emotion data in real time.

[1675] Step 8:

[1676] The user's emotion data generated by the terminal is transmitted to the server.

[1677] Step 9:

[1678] The server combines the analysis results of the transcription data with the user's emotional data to evaluate the customer's emotions and reactions. For example, it may evaluate a comment such as "it's too expensive" as negative and determine that the user's emotions are impatience or confusion.

[1679] Step 10:

[1680] The server generates an appropriate talk script based on the evaluation results, for example, "Price is certainly an important point, but our products offer excellent long-term cost performance."

[1681] Step 11:

[1682] The server sends the generated talk script to the terminal.

[1683] Step 12:

[1684] The terminal presents the generated talk script to the user visually or audibly.

[1685] Step 13:

[1686] The user refers to the provided talk script and replies, "Price is important, but this product offers excellent long-term cost performance," and continues the conversation.

[1687] Step 14:

[1688] The device records the audio again and repeats the process from step 3 onwards, generating new transcripts sequentially and sending them to the server, where the process of analysis, evaluation and script generation continues.

[1689] In this way, this system supports the progress of meetings in real time and enables appropriate responses to be made taking into account the feelings of both the user and the customer.

[1690] Example 2

[1691] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1692] Conventional meeting support systems were unable to transcribe voice data in real time or perform subsequent analysis quickly and accurately. Furthermore, it was difficult to generate appropriate talk scripts that took into account the customer's emotions and reactions, placing a heavy burden on sales representatives. Furthermore, there were insufficient means to dynamically respond to the progress of the meeting and the user's emotions, making it difficult to conduct effective sales activities.

[1693] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring conference audio in real time and converting it into text data, means for transmitting the text data to the server at regular intervals, means for analyzing the text data using natural language processing technology, means for analyzing the emotions of the user and the customer in real time using an emotion analysis engine, means for integrating the emotion data of the customer and the user and evaluating the customer's reaction, means for generating a talk script based on the evaluation result, and means for presenting the generated talk script to the user. This enables real-time analysis of audio data and rapid generation of a talk script according to the customer's emotion.

[1694] "Means for capturing conference audio in real time and converting it into text data" refers to a device or system that includes technology for instantly capturing audio generated during a conference as digital data and converting that audio data into text data.

[1695] The "means for transmitting character data to a server at regular intervals" refers to a device or system for transmitting the generated character data to a server at appropriate timing.

[1696] "Means for analyzing text data using natural language processing techniques" refers to a device or system that uses natural language processing algorithms and techniques to perform text analysis on acquired text data.

[1697] "Means for analyzing user and customer emotions in real time using an emotion analysis engine" refers to a device or system that includes technology for analyzing the voice tone and facial expressions of users and customers and evaluating their emotions in real time.

[1698] "Means for integrating emotional data of customers and users and evaluating customer reactions" refers to a device or system that includes technology for integrating emotional data obtained from users and customers and evaluating customer reactions based on the results.

[1699] The "means for generating a talk script based on the evaluation results" refers to a device or system including a technology for generating a talk script including appropriate conversation content based on the customer's reaction evaluation results.

[1700] The "means for presenting the generated talk script to the user" refers to a device or system that includes a technique for visually or audibly presenting the generated talk script to the user.

[1701] A "generative AI model" is an artificial intelligence algorithm trained using large datasets, and is a model for generating and analyzing text data.

[1702] A "prompt" is an instruction entered into a generative AI model, and is text used to control the AI's behavior and generation.

[1703] The present invention provides a system for supporting salespeople in responding to customers promptly and appropriately in meetings and business negotiations. Specific embodiments for carrying out the present invention will be described below.

[1704] This system is broadly composed of two main components: a terminal and a server.

[1705] Device configuration

[1706] The terminal is a device used in a meeting or business negotiation, and specifically includes a microphone, a camera, a voice recognition API, and an emotion analysis API.

[1707] 1. Acquire audio data:

[1708] The device uses a built-in microphone to collect audio data during the meeting, and audio capture is performed in real time.

[1709] 2. Audio data conversion:

[1710] The device converts the collected voice data into text data using the Google Speech-to-Text API or IBM Watson Speech to Text API.

[1711] 3. Emotion recognition:

[1712] The device uses a camera to capture the user's facial expressions and a microphone to capture voice tones. This data is then analyzed using emotion analysis APIs such as Microsoft Azure Emotion API and Amazon Rekognition to generate emotion data in real time.

[1713] 4. Data transmission:

[1714] The terminal buffers the generated character data and emotion data at regular intervals and transmits them to the server.

[1715] Server Configuration

[1716] The server is a system that receives data sent from the device, analyzes it, and generates a talk script, using natural language processing technology and generative AI models.

[1717] 1. Receiving data:

[1718] The server receives the character data and emotion data sent from the terminal.

[1719] 2. Data Analysis:

[1720] The server uses natural language processing technology (for example, Python's NLTK library or SpaCy) to analyze the received text data and evaluate the intent and sentiment of the customer's comments.

[1721] 3. Emotional data integration:

[1722] The server integrates the received emotional data and evaluates the customer's reaction comprehensively.

[1723] 4. Generate the talk script:

[1724] The server uses a generative AI model (e.g., OpenAI's GPT-3) based on the analysis results to generate an appropriate conversation script, which provides the optimal response method for the user.

[1725] 5. Sending the talk script:

[1726] The server transmits the generated talk script to the terminal, which then presents the talk script to the user visually or audibly.

[1727] Specific examples

[1728] Initial Setup and Starting a Meeting

[1729] The user starts the system and sets the goal of the meeting as "concluding a contract for a new product." The device then activates its voice recognition function and emotion engine and begins recording the audio of the meeting.

[1730] Real-time transcription and analysis example

[1731] A customer says, "I think the price of this product is too high." The device transcribes this speech in real time and sends the text data to the server. The server analyzes the text data and evaluates the customer's sentiment as negative.

[1732] Example of user emotion recognition

[1733] The device uses an emotion engine to recognize emotions such as impatience or confusion from the user's tone of voice and facial expressions, and then transmits the user's emotion data to the server.

[1734] Example of generating and presenting a talk script

[1735] The server generates a script that says, "Price is certainly an important point, but our product offers excellent long-term cost performance." The server sends the generated script to the terminal, which then displays it to the user.

[1736] Example of user response

[1737] The user responds, "Price is important, and this product offers great long-term value for money," and continues the conversation with the customer.

[1738] Prompt Sentence Examples

[1739] "Provide a script to alleviate the feelings of impatience identified by the emotion engine."

[1740] "How can I generate a talk script to respond to negative comments from customers?"

[1741] This system enables sales representatives to respond quickly and appropriately during meetings and negotiations with customers, enabling them to conduct effective sales activities.

[1742] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1743] Step 1: Acquire and transcribe audio data

[1744] Input: A user starts a conference and a microphone collects audio.

[1745] How it works: The device captures real-time audio during meetings, which is then sent to the Google Speech-to-Text API or IBM Watson Speech to Text API for conversion into text data.

[1746] Data processing: The audio data is converted into text data. Specifically, the API analyzes the audio waveform and generates the corresponding text.

[1747] Output: Text data (e.g., "This product is too expensive").

[1748] Step 2: Sending character data

[1749] Input: The character data generated in step 1.

[1750] Operation: The terminal buffers the generated character data at a fixed time-slide interval and transmits it to the server sequentially.

[1751] Data processing: Data is accumulated through buffering and converted into a format that can be sent in fixed amounts.

[1752] Output: Character data sent to the server.

[1753] Step 3: Parsing character data

[1754] Input: Character data sent to the server.

[1755] How it works: The server analyzes text data using Python's NLTK library, SpaCy, etc. Specifically, it performs semantic and sentiment analysis of the text.

[1756] Data processing: Grammatical analysis, keyword extraction, and sentiment analysis are performed on the text data.

[1757] Output: Analysis results (e.g., "negative sentiment", "price mentions").

[1758] Step 4: Emotion recognition and data transmission

[1759] Input: User's voice tone and facial expression data during the meeting.

[1760] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice tone, and this data is analyzed in real time using tools like the Microsoft Azure Emotion API and Amazon Rekognition.

[1761] Data processing: The captured voice tone and facial expression data are converted into emotion data through emotion analysis algorithms.

[1762] Output: Generated user emotion data (e.g., "impatience").

[1763] Step 5: Assess your emotions and reactions

[1764] Input: Analysis results of text data and emotion data.

[1765] Operation: The server integrates the results of the character data analysis with the user's emotional data to evaluate the customer's reaction. Specifically, it compares both sets of emotional data and performs a comprehensive emotional evaluation.

[1766] Data processing: By integrating text data and emotion data, more detailed emotion evaluation is performed.

[1767] Output: Customer reaction evaluation results (e.g., "Customers reacted negatively to the price").

[1768] Step 6: Generate the talk script

[1769] Input: Customer response evaluation results.

[1770] Operation: The server uses the generative AI model to generate an appropriate talk script based on the evaluation results. The server inputs the prompt sentence into the generative AI model and obtains the script.

[1771] Data processing: Generative AI models generate natural-sounding conversation scripts based on context.

[1772] Output: Generated talk script (e.g., "Price is important, but this product offers great long-term value for money").

[1773] Step 7: Send and present your talk script

[1774] Input: The generated talk script.

[1775] Operation: The server sends the generated talk script to the terminal, which receives it and presents it to the user visually or audibly.

[1776] Data processing: The talk script is converted into a format that is easy for the user to view or to use in audio.

[1777] Output: The talk script presented to the user (e.g., the script displayed on the device screen).

[1778] Through this detailed step-by-step process, the present invention helps users respond quickly and appropriately during meetings and business negotiations with customers.

[1779] (Application example 2)

[1780] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1781] It is important for sales representatives to respond quickly and appropriately to customer emotions and reactions during meetings and negotiations. However, it is difficult to understand the content of the conversation in real time, instantly grasp the customer's emotions, and provide an appropriate dialogue script that matches them. There is a need to solve this problem and significantly improve the efficiency and effectiveness of sales activities. Furthermore, to achieve real-time support, it is necessary to operate an integrated system on a device worn by the sales representative.

[1782] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1783] In this invention, the server includes means for transcribing conference audio data in real time, means for analyzing the transcribed data and evaluating the customer's emotions and reactions, means for generating an appropriate talk script based on the customer's emotions and reactions, means for presenting the generated talk script to a user, and means for displaying the generated talk script in real time on a device worn by a sales representative. This enables the sales representative to respond appropriately on the spot during a conversation with a customer, thereby increasing the success rate of meetings and business negotiations.

[1784] "Conference audio data" refers to all audio information spoken during the conference, including conversations between the sales representative and the customer.

[1785] "Real-time transcription means" refers to a technical means for instantly converting collected audio data into text data.

[1786] "Transcription data" means audio data converted into text format.

[1787] "Means for analyzing the transcript data" refers to the technical means for analyzing the converted transcript data to understand and classify its content.

[1788] "Means for assessing emotions and reactions" refers to technological means for determining customer emotions and reactions based on analyzed transcript data.

[1789] A "talk script" is a document that provides instructions and examples of conversations that a sales representative can use to respond appropriately to a customer.

[1790] "Generative means" refers to technological means that use engines or algorithms to create new information or data for a specific purpose.

[1791] "Means for presenting" refers to a technical means for displaying or notifying the generated talk script to the sales representative.

[1792] "Means for displaying on a device in real time" refers to a technical means for instantly providing the generated talk script on the display screen of a device (such as smart glasses) worn by a sales representative.

[1793] (Embodiment of the Invention)

[1794] The present invention provides a system that enables salespeople to respond quickly and appropriately during meetings and negotiations with customers, supporting effective sales activities. The system transcribes voice data in real time, analyzes the transcript to evaluate the customer's emotions and reactions, generates an appropriate talk script based on the analysis, and displays it in real time on a device worn by the salesperson.

[1795] System Configuration

[1796] 1. Terminal

[1797] The device (e.g., smart glasses) performs the following operations:

[1798] Audio data collection: The built-in microphone of the smart glasses is used to record the audio of the meeting in real time.

[1799] Transcription: Recorded audio data is transcribed in real time using the Google Cloud Speech-to-Text API.

[1800] Emotion recognition: The smart glasses' built-in camera captures the customer's facial expressions and uses the Microsoft Azure Face API to analyze their emotions in real time.

[1801] Data transmission: The buffered transcription data and emotion data are sent to the server.

[1802] 2. Server

[1803] The server performs the following operations:

[1804] Data reception: Receives transcription data and emotion data sent from the device.

[1805] Natural Language Processing: Analyze the received transcription data using natural language processing techniques (e.g., NLTK with Sentiment Analysis).

[1806] Sentiment assessment: Evaluate customer emotions and reactions based on the analysis results.

[1807] Talk script generation: Use a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[1808] Send data: Send the generated talk script to the device.

[1809] 3. User (Salesperson)

[1810] Salespeople use the system as follows:

[1811] Check the talk script: Refer to the talk script displayed on the terminal and proceed with the conversation with the customer.

[1812] Real-time response: Based on the generated conversation script, appropriate responses are provided according to the customer's emotions.

[1813] Specific examples

[1814] Initial Setup and Conference Start Example

[1815] A sales representative puts on the smart glasses and starts the system. The purpose of the meeting is set to "conclude a contract for a new product."

[1816] The smart glasses activate their voice recognition and emotion recognition engines and begin recording and capturing the meeting audio and customer facial expressions in real time.

[1817] Transcription and analysis examples

[1818] A customer says, "I think this product is priced too high."

[1819] The smart glasses transcribe this speech in real time and send the text data to a server.

[1820] The server analyzes the text data and evaluates the customer's sentiment as negative.

[1821] Examples of prompt statements

[1822] Audio Transcription: 'I think the price of this product is too high.'

[1823] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[1824] Generate the appropriate talk script.

[1825] As described above, the present invention provides a system that supports the progress of a conference in real time and supports the responses of sales representatives.

[1826] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1827] Step 1:

[1828] Audio data collection

[1829] The device (smart glasses) uses a built-in microphone to record the audio of the meeting in real time.

[1830] Input: Audio data during the meeting

[1831] Output: Recorded audio data

[1832] Specifically, the system records the conversation between the sales representative and the customer in high quality and temporarily stores it in local memory for post-processing.

[1833] Step 2:

[1834] Transcription

[1835] The device transcribes the recorded audio data in real time using the Google Cloud Speech-to-Text API.

[1836] Input: Recorded audio data

[1837] Output: Real-time transcribed text data

[1838] Specifically, audio data is sent to the API, and the resulting text data is buffered and recorded sequentially.

[1839] Step 3:

[1840] emotion recognition

[1841] The device captures the customer's facial expressions using its built-in camera, and analyzes this video data in real time using the Microsoft Azure Face API to assess the customer's emotions.

[1842] Input: Customer facial expression video data

[1843] Output: Real-time analyzed emotion data

[1844] Specifically, the device's camera captures video of the customer at regular intervals, sends the video to an API, and obtains emotion recognition results.

[1845] Step 4:

[1846] Sending data

[1847] The terminal transmits the buffered transcription data and real-time emotion data to the server at regular time-sliding intervals.

[1848] Input: Transcription text data, emotion data

[1849] Output: Send data to the server

[1850] Specifically, the buffered data is collected into packets at regular intervals and sent to a specified address on the server.

[1851] Step 5:

[1852] Natural Language Processing

[1853] The server analyzes the received transcription data using natural language processing technology (e.g., NLTK with Sentiment Analysis) to understand and classify the content of the conversation.

[1854] Input: Transcribed text data sent to the server

[1855] Output: Analysis data after natural language processing

[1856] Specifically, the text data is tokenized, and the context and sentiment of each token is analyzed to determine the meaning and sentiment of the customer's words.

[1857] Step 6:

[1858] Emotional evaluation

[1859] The server evaluates the customer's emotions and reactions based on the analysis results, which also incorporates the received emotional data.

[1860] Input: Analysis data after natural language processing, emotion data

[1861] Output: Evaluated customer sentiment and reaction data

[1862] Specifically, analytical data and emotional data are combined to determine and evaluate the actual emotions felt by the customer.

[1863] Step 7:

[1864] Generate a talk script

[1865] The server uses a generative AI model such as OpenAI's GPT-4 to generate an appropriate talk script based on the evaluation results.

[1866] Input: Evaluated customer sentiment and reaction data

[1867] Output: Generated talk script

[1868] Specifically, the prompt sentence is input into the generative AI model, and the obtained talk script is organized and saved.

[1869] Specific prompt examples:

[1870] Audio Transcription: 'I think the price of this product is too high.'

[1871] Customer emotions: { 'Anger': 0.1, 'Contempt': 0.0, 'Disgust': 0.1, 'Fear': 0.0, 'Happiness': 0.0, 'Neutral': 0.4, 'Sadness': 0.0, 'Surprise': 0.1}

[1872] Generate the appropriate talk script.

[1873] Step 8:

[1874] Sending data

[1875] The server transmits the generated talk script to the terminal.

[1876] Input: Generated talk script

[1877] Output: Sending data to the terminal

[1878] Specifically, the generated talk script is sent to the specified address of the terminal.

[1879] Step 9:

[1880] Show talk script

[1881] The terminal displays the generated talk script in real time within the field of view of the salesperson.

[1882] Input: Talk script received from the server

[1883] Output: Talk script displayed to sales representatives

[1884] Specifically, the talk script will be visually displayed on the device display, allowing sales representatives to quickly refer to it.

[1885] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1886] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1887] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1888] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1889] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1890] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1891] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1892] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1893] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1894] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1895] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1896] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1897] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1898] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1899] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1900] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1901] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1902] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1903] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1904] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1905] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1906] The following is further disclosed regarding the above embodiment.

[1907] (Claim 1)

[1908] A means of transcribing meeting audio data in real time,

[1909] A means of analyzing the transcript data to assess customer sentiment and reactions;

[1910] A means to generate appropriate conversation scripts based on customer emotions and reactions,

[1911] means for presenting the generated talk script to a user;

[1912] A system including:

[1913] (Claim 2)

[1914] 2. The system according to claim 1, further comprising means for continuously analyzing the progress of the conference and dynamically updating the talk script according to the customer's enthusiasm.

[1915] (Claim 3)

[1916] 10. The system of claim 1, further comprising means for analyzing the transcription data using natural language processing techniques.

[1917] "Example 1"

[1918] (Claim 1)

[1919] means for acquiring audio data of a conference in real time using an audio input device;

[1920] A means of transcribing in real time using a speech recognition API;

[1921] A means for sequentially buffering the transcription data and transmitting it to a server at regular time-slide intervals;

[1922] A means for receiving the transcription data in a server and analyzing it using natural language processing technology;

[1923] A means of assessing customer emotions and reactions based on the analysis results,

[1924] A means for generating an appropriate talk script using a generative AI model based on the evaluation results; and

[1925] means for presenting the generated talk script to a user;

[1926] A system including:

[1927] (Claim 2)

[1928] 2. The system according to claim 1, further comprising means for continuously analyzing the progress of the conference and dynamically updating the talk script according to the customer's enthusiasm.

[1929] (Claim 3)

[1930] 10. The system of claim 1, further comprising means for analyzing the transcription data using natural language processing techniques.

[1931] "Application Example 1"

[1932] (Claim 1)

[1933] A means of transcribing meeting audio data in real time,

[1934] A means of analyzing the transcript data to assess customer sentiment and reactions;

[1935] A means to generate appropriate conversation scripts based on customer emotions and reactions,

[1936] means for presenting the generated talk script to a user;

[1937] A means of collecting voice data during sales negotiations or customer interactions and providing feedback and responses generated based on the analysis results in real time;

[1938] A system including:

[1939] (Claim 2)

[1940] A means to analyze the progress of the meeting in real time and dynamically update the talk script according to the customer's enthusiasm,

[1941] The system according to claim 1, which provides appropriate feedback during business negotiations or customer interactions.

[1942] (Claim 3)

[1943] a means for analyzing the transcription data using natural language processing techniques;

[1944] 10. The system of claim 1, further comprising: means for generating an appropriate response using a generative AI model.

[1945] "Example 2: Combining Emotion Engines"

[1946] (Claim 1)

[1947] A means to capture the audio of the meeting in real time and convert it into text data,

[1948] means for transmitting character data to a server at regular intervals;

[1949] A means for analyzing character data using natural language processing technology;

[1950] A means to analyze user and customer sentiment in real time using a sentiment analysis engine;

[1951] a means of integrating customer and user sentiment data and assessing customer responses;

[1952] means for generating a talk script based on the evaluation result;

[1953] means for presenting the generated talk script to a user;

[1954] A system including:

[1955] (Claim 2)

[1956] 2. The system according to claim 1, further comprising means for continuously monitoring the progress of the conference and dynamically updating the talk script in response to the reactions of the guests and the emotions of the users.

[1957] (Claim 3)

[1958] 10. The system of claim 1, comprising means for using a generative AI model to analyze the generated text data.

[1959] "Application example 2 when combining emotion engines"

[1960] (Claim 1)

[1961] A means of transcribing meeting audio data in real time,

[1962] A means of analyzing the transcript data to assess customer sentiment and reactions;

[1963] A means to generate appropriate conversation scripts based on customer emotions and reactions,

[1964] means for presenting the generated talk script to a user;

[1965] a means for displaying the information in real time on a device worn by the sales representative;

[1966] A system including:

[1967] (Claim 2)

[1968] 2. The system according to claim 1, further comprising means for continuously analyzing the progress of the conference and dynamically updating the talk script according to the customer's enthusiasm.

[1969] (Claim 3)

[1970] 10. The system of claim 1, further comprising means for analyzing the transcription data using natural language processing techniques. [Explanation of symbols]

[1971] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of transcribing meeting audio data in real time, A means of analyzing the transcript data to assess customer sentiment and reactions; A means to generate appropriate conversation scripts based on customer emotions and reactions, means for presenting the generated talk script to a user; A system including:

2. 2. The system according to claim 1, further comprising means for continuously analyzing the progress of the conference and dynamically updating the talk script in accordance with the enthusiasm of the customers.

3. The system of claim 1 , further comprising means for analyzing the transcription data using natural language processing techniques.

Images