System

Abstract

To provide a system.SOLUTION: A system comprises: means of imaging a roll play moving image of a store staff; means of uploading the imaged moving image to a server; means of analyzing words, an attitude, or a response of a guest within the moving image; means of analyzing a feeling state of the store staff; generation system AI means of generating a proposed talk on the basis of a result of feeling analysis of the store staff; and means of confirming the generated proposed talk, and utilizing it for actual reception.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] One issue is that store staff are not receiving enough guidance or feedback on how to deliver effective proposal talks when serving customers. In particular, there are limited opportunities for other staff to learn from the proposal talks of highly successful staff. [Means for solving the problem]

[0005] This invention analyzes role-playing videos of store staff, who then ask questions based on the analysis results. Generative AI then generates effective proposal speeches in response to those questions, helping to improve the customer service skills of store staff. Furthermore, the generative AI learns the proposal speeches of highly successful staff and provides feedback to store staff based on the learning results. This makes it possible to efficiently provide guidance and feedback to store staff to help them use effective proposal speeches when serving customers. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] This embodiment of the present invention uses video analysis technology as a means of analyzing role-playing videos of store staff. Specifically, store staff film role-playing videos simulating actual customer service scenes, and input the videos into a generative AI. The generative AI then analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions.

[0029] "Example 2"

[0030] Next, an interactive interface is provided as a means for store staff to ask questions to the generative AI. Specifically, store staff can ask the generative AI questions such as, "What is the best suggestion to make in this situation?"

[0031] "Example 3"

[0032] The generative AI then uses natural language generation technology to generate effective suggestion speeches. Specifically, the generative AI generates appropriate suggestion speeches in response to questions from store staff and presents them to them.

[0033] "Example 4"

[0034] Furthermore, machine learning technology is used to train the generative AI to learn the proposal speech of highly successful store staff. Specifically, the generative AI learns the proposal speech of highly successful store staff, and the results are used to generate new proposal speech.

[0035] "Example 5"

[0036] Finally, the interactive interface is used again to provide feedback to store staff based on the proposed speech learned by the generative AI. Specifically, the system provides specific advice and guidance to store staff based on the proposed speech generated by the generative AI.

[0037] The processing flow of each embodiment will be described below.

[0038] "Example 1"

[0039] Step 1: Store staff film role-playing videos simulating actual customer service situations.

[0040] Step 2: Input the filmed role-play video into the generative AI.

[0041] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[0042] "Example 2"

[0043] Step 1: Store staff ask questions to the generative AI through an interactive interface, such as, "What is the best suggestion to make in this situation?"

[0044] "Example 3"

[0045] Step 1: The generative AI generates an appropriate suggestion speech in response to the store staff's question. Step 2: The generated suggestion speech is presented to the store staff.

[0046] "Example 4"

[0047] Step 1: Have the generative AI learn the proposal talk of highly successful store staff.

[0048] Step 2: The generative AI uses the learning results to generate new proposed talks.

[0049] "Example 5"

[0050] Step 1: Based on the proposed conversation generated by the generative AI, specific advice and guidance is provided to store staff.

[0051] Example 1

[0052] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0053] Conventional training methods for improving store staff's customer service skills make it difficult to accurately evaluate the staff's actual customer service attitude and customer reactions, making it impossible to provide effective feedback.In addition, there was a lack of concrete measures to improve the quality of staff's suggestion talk, which limited the improvement in customer service skills.

[0054] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0055] In this invention, the server includes means for shooting role-playing videos of store staff, means for uploading the shot videos to the server, means for receiving the uploaded videos and converting the speech into text using speech recognition technology, means for analyzing the text data and video data in the videos using a generative AI model, means for evaluating the customer service skills of the store staff based on the analysis results and generating feedback, and means for displaying the generated feedback to the user. This makes it possible to accurately evaluate the customer service attitude of the store staff and the reactions of customers and provide effective feedback.

[0056] "Store staff" refers to employees who serve customers in a store.

[0057] "Role-play videos" are videos filmed by store staff simulating actual customer service situations.

[0058] A "server" is a computer system that receives, processes, stores, and transmits data over a network.

[0059] "Voice recognition technology" refers to technology that converts voice data into text data.

[0060] A "generative AI model" is a model that uses artificial intelligence technology to analyze data and generate results.

[0061] "Text data" refers to text information converted using voice recognition technology.

[0062] "Video data" refers to image information obtained from moving images.

[0063] "Analysis" is the process of examining data in detail and extracting meaning and patterns.

[0064] "Feedback" refers to evaluations and advice provided based on the analysis results.

[0065] "Users" refer to store staff and managers who use this system.

[0066] MODE FOR CARRYING OUT THE INVENTION

[0067] The present invention is a system for improving the customer service skills of store staff, which analyzes role-playing videos and provides feedback. A specific embodiment of this system will be described below.

[0068] System configuration

[0069] Hardware and Software Configuration

[0070] 1. User's device

[0071] Photography devices such as smartphones and digital cameras

[0072] An application for uploading videos to a server

[0073] 2. Server

[0074] A computer system that receives, processes, and analyzes video

[0075] Speech recognition technology (e.g. Google (registered trademark) Cloud Speech-to-Text API)

[0076] Generative AI models (e.g., OpenAI's GPT-4)

[0077] Recording and uploading videos

[0078] Users use their smartphones or digital cameras to record role-playing videos of store staff. For example, they can record a scene in which a staff member greets a customer by saying "Welcome." After filming, users select the video file through an application on their device and upload it to the server.

[0079] Video reception and preprocessing

[0080] The server receives the video file uploaded from the device. After receiving it, it checks the format and size of the video file and performs format conversion or compression if necessary. Next, the server extracts the audio portion of the video and converts it to text using speech recognition technology, such as the Google Cloud Speech-to-Text API.

[0081] Video Analysis

[0082] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text and video data in the video. Specifically, it performs the following processes:

[0083] Text data analysis: The server uses natural language processing technology to analyze the text data converted by speech recognition technology and evaluates the content of the store staff's speech. For example, it analyzes the tone and context of the phrase "Welcome."

[0084] Video data analysis: The server uses image recognition technology to analyze the video data and evaluate the facial expressions and gestures of staff and the customer's reactions. For example, it analyzes whether the staff is smiling and the customer's satisfaction level.

[0085] Results generation and feedback

[0086] The server evaluates the customer service skills of store staff based on the analysis results and generates feedback on areas for improvement and merit. For example, it generates specific advice such as, "You don't smile often, so try to smile more." The generated feedback is saved in text format and sent to the device.

[0087] View Feedback

[0088] The terminal displays the feedback received from the server to the user. For example, the evaluation results may be displayed in text or graph format on the application screen. The user can check the displayed feedback and use it to improve their next customer service experience.

[0089] Examples of specific examples and prompts

[0090] Examples:

[0091] Users film role-playing videos on their smartphones and upload them to the server via the application.

[0092] The server receives the video and converts the audio portion into text using the Google Cloud Speech-to-Text API.

[0093] The server uses a generative AI model (OpenAI's GPT-4) to analyze the text and video data, for example evaluating the tone of the staff member's "Welcome" remark and the customer's facial expression.

[0094] Based on the analysis results, the server generates feedback such as "You're not smiling enough, so try to smile more," and sends it to the device.

[0095] The device displays the feedback to the user, who can use it to improve their next customer service experience.

[0096] Example prompt sentence:

[0097] "Please analyze the customer service attitude of the store staff in this video and provide feedback on areas for improvement."

[0098] "Please evaluate the comments made by staff in the video and the reactions of customers, and provide advice to help improve customer service skills."

[0099] In this way, a system is realized in which the server, terminals, and users work together to improve the customer service skills of store staff.

[0100] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0101] Step 1:

[0102] Recording and uploading videos

[0103] The user uses a smartphone or digital camera to record a role-playing video of a store staff member, for example, greeting a customer by saying "Welcome."

[0104] Input: Filmed roleplay video file

[0105] The device provides an application for uploading the captured video to the server. The user selects a video file through the application on the device and presses the upload button.

[0106] Output: Video file uploaded to the server

[0107] Step 2:

[0108] Video reception and preprocessing

[0109] The server receives the video file uploaded from the device, checks the format and size of the video file, and performs format conversion or compression as necessary.

[0110] Input: Uploaded video file

[0111] The server extracts the audio portion of the video and converts the audio into text using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[0112] Output: Text data (text information converted from audio)

[0113] Step 3:

[0114] Video Analysis

[0115] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text and visual data in the video.

[0116] Input: Text data, video data

[0117] The server analyzes the text data using natural language processing technology and evaluates the content of the store staff's comments, such as the tone and context of the phrase "Welcome."

[0118] The server analyzes the video data using image recognition technology to evaluate the facial expressions and gestures of staff and the customer's reactions, for example, whether the staff is smiling or not and the customer's satisfaction level.

[0119] Output: Analysis results (evaluation of speech content, facial expressions and gestures, evaluation of customer reactions)

[0120] Step 4:

[0121] Results generation and feedback

[0122] The server evaluates the customer service skills of store staff based on the analysis results and generates feedback on areas for improvement and strengths. For example, it generates specific advice such as, "You don't smile often, so try to smile more."

[0123] Input: Analysis results

[0124] The server saves the generated feedback in text format and transmits it to the terminal.

[0125] Output: Feedback (textual evaluation and advice)

[0126] Step 5:

[0127] View Feedback

[0128] The terminal displays the feedback received from the server to the user, for example, by displaying the evaluation results in text or graph format on the application screen.

[0129] Input: Feedback

[0130] The user can check the displayed feedback and use it to improve their next customer service experience.

[0131] Output: Feedback displayed to the user (evaluation results and advice)

[0132] In this way, a system is realized in which the server, terminals, and users work together to improve the customer service skills of store staff.

[0133] (Application example 1)

[0134] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Traditional store staff training methods lack the feedback needed to effectively improve their customer service skills. Furthermore, there are limited ways for staff to objectively evaluate their own customer service attitudes and language, making self-improvement difficult. Furthermore, the time and cost required for staff training are also major issues.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0137] In this invention, the server includes a means for analyzing role-playing videos of store staff, a means for store staff to ask questions based on the analysis results, a generative AI means for generating effective suggestion talk in response to the questions, an application means installed on a smartphone, a means for converting video frames into text, and a means for inputting prompts into the generative AI to obtain the analysis results. This allows store staff to objectively evaluate their customer service skills and identify specific areas for improvement. It also improves training efficiency and reduces costs.

[0138] "Store staff" refers to employees who provide products and services to customers in physical stores.

[0139] "Role-play video" is a video that shows store staff simulating customer service.

[0140] The "analysis method" refers to technology that analyzes role-play videos and evaluates the language and attitude of store staff and customer reactions.

[0141] "Generative AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of generating proposed talks.

[0142] The "application means installed on a smartphone" is software that runs on a smartphone and is an application for supporting training of store staff.

[0143] "Means for converting video frames into text" refers to a technology that analyzes each frame of video and converts its content into text format.

[0144] A "prompt" is an instruction to be input to a generative AI, and is text that includes the content of a video frame.

[0145] The "means of obtaining analysis results" refers to the technology of inputting a prompt sentence into a generative AI and obtaining the resulting analysis data.

[0146] The system for implementing this invention analyzes role-playing videos of store staff and provides feedback to improve the staff's customer service skills. The system is implemented using an application installed on a smartphone.

[0147] First, store staff use their smartphones to film a role-playing video. The video is then uploaded to a server via an application. The server analyzes the video frame by frame and converts the content of each frame into text. This text is used as a prompt to input to the generative AI.

[0148] Based on the prompt text, the generative AI analyzes the language and attitude of store staff in the video, as well as customer reactions. The analysis results are provided as specific feedback on the staff's customer service skills, including areas for improvement and specific advice.

[0149] The hardware used is a smartphone, and the software uses Python, OpenCV, and OpenAI API. Specifically, OpenCV is used to divide the video into frames and convert the content of each frame into text. The OpenAI API is used to input prompt text into the generative AI and obtain the analysis results.

[0150] As a concrete example, the following prompt sentence is input into the generative AI.

[0151] Analyze the following video frame:

[0152] Frame 1: Staff members are smiling and greeting customers.

[0153] Frame 2: A staff member explains the product.

[0154] Frame 3: The customer is asking a question.

[0155] Frame 4: Staff member answers questions.

[0156] Frame 5: The customer nods approvingly.

[0157] By inputting this prompt into generative AI, specific feedback on the staff's customer service skills can be obtained. This allows store staff to objectively evaluate their own customer service skills and identify specific areas for improvement. It also leads to more efficient training and reduced costs.

[0158] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0159] Step 1:

[0160] The user shoots a role-playing video using a smartphone. The video is saved in the smartphone. The input is the role-playing video shot by the user, and the output is the video file saved in the smartphone.

[0161] Step 2:

[0162] The device uploads the captured role-play video to a server via the application. The input is the video file stored on the smartphone, and the output is the video file uploaded to the server.

[0163] Step 3:

[0164] The server splits the uploaded video into frames. The input is the video file uploaded to the server, and the output is the image data of each split frame. Specifically, the video is split into frames using OpenCV.

[0165] Step 4:

[0166] The server converts the content of each frame into text. The input is the image data of each divided frame, and the output is text data describing the content of each frame. Specifically, the server uses image recognition technology to analyze the content of the frame and convert it into text format.

[0167] Step 5:

[0168] The server generates a prompt sentence to be input to the generative AI. The input is text data describing the content of each frame, and the output is the generated prompt sentence. Specifically, the text data of each frame is concatenated to create the prompt sentence.

[0169] Step 6:

[0170] The server inputs a prompt to the generative AI and obtains the analysis results. The input is the generated prompt, and the output is the analysis results obtained from the generative AI. Specifically, the server inputs the prompt using OpenAI's API and obtains the analysis data.

[0171] Step 7:

[0172] The server generates feedback for store staff based on the analysis results. The input is the analysis results obtained from the generative AI, and the output is specific feedback content. Specifically, the server analyzes the analysis results and creates feedback that includes areas for improvement and advice regarding the staff's customer service skills.

[0173] Step 8:

[0174] The device receives feedback from the server and displays it to the user. The input is the feedback sent from the server, and the output is the feedback displayed on the smartphone screen. Specifically, the feedback is received through an application and visually presented to the user.

[0175] Example 2

[0176] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] Traditional store staff training methods required a lot of time and effort for staff to master effective sales pitches. They also lacked immediate feedback to help staff deliver appropriate sales pitches in real-life situations. This slowed down staff performance improvement and potentially led to lower customer satisfaction.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0179] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, means for the store staff to input questions to the generative AI through an interface, means for sending the input questions to the generative AI model, and means for receiving answers from the generative AI model and displaying them to the store staff. This enables store staff to quickly and effectively learn appropriate suggestion talk and immediately use it in actual situations.

[0180] "Store staff" refers to employees who deal with customers and make product suggestions in stores.

[0181] "Role-play video" refers to footage of store staff simulating customer interactions and product suggestions.

[0182] "Means of analysis" refers to the techniques and methods used to analyze role-play videos and evaluate staff performance and the effectiveness of proposal talks.

[0183] "Means for asking questions" refers to the interface or method that store staff use to input questions about specific situations to the generative AI.

[0184] "Generative AI means" refers to artificial intelligence technology that generates appropriate suggestion talk in response to an input question.

[0185] "Interface" refers to the user interface that store staff use to input questions into the generative AI.

[0186] "Generative AI model" refers to an artificial intelligence model that generates answers based on input prompts.

[0187] "Means of transmission" refers to the communication technology or method for transmitting the input question to the generative AI model.

[0188] "Means for displaying" refers to the technology or method for visually presenting the answers received from the generative AI model to store staff.

[0189] MODE FOR CARRYING OUT THE INVENTION

[0190] This invention is a system that uses generative AI to help store staff learn effective proposal talk. The system works in cooperation with three parties: a server, a terminal, and a user.

[0191] Hardware and software used

[0192] The server was developed using Python and implemented as a web application using the Flask framework. The generative AI model used was OpenAI's GPT-4. The terminals used by store staff were PCs or tablets, which accessed the server using a web browser (e.g., GOOGLE CHROME (registered trademark)).

[0193] Program processing

[0194] The server hosts a web application using the Flask framework and provides an interface for receiving questions from store staff. The interface uses HTML and JavaScript® to create a user-friendly form that allows store staff to enter their questions.

[0195] The terminal (store staff's PC or tablet) uses a web browser to access the server and display the interface. The user (store staff) inputs questions to the generative AI through the interface. For example, they might input a question such as, "What is the best suggestion to make in this situation?"

[0196] The entered question is sent to the server via JavaScript. The server passes the received question to the generative AI model. The generative AI model receives the question as a prompt sentence and generates an appropriate answer. The server returns the answer received from the generative AI model to the device, and the device displays the received answer on the interface.

[0197] Specific examples

[0198] For example, consider the case where a store staff member asks, "What is the best suggestion phrase to use when introducing a new product?" This question is sent to the generative AI model as a prompt sentence like the following:

[0199] Example prompt sentence:

[0200] What is the best pitch for introducing a new product?

[0201] The generative AI model generates appropriate suggested speech based on this prompt and sends it back to the server, which then sends the generated suggested speech back to the device and displays it on the interface for the user to review.

[0202] This system enables store staff to quickly and effectively learn appropriate sales pitches and immediately use them in real-life situations.

[0203] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0204] Step 1:

[0205] The user accesses the interface

[0206] The user (store staff) opens a web browser (e.g., Google Chrome) and accesses a specified URL (e.g., http: / / example.com). The server serves the web page using the Flask framework. The input is the user's action of accessing the URL, and the output is the web page on which the interface is displayed.

[0207] Step 2:

[0208] The user enters a question

[0209] The user enters "What is the best suggested talk for this situation?" into the text box on the web page and clicks the submit button. The input is the user's action of entering a question into the text box, and the output is the entered question data.

[0210] Step 3:

[0211] The device sends a question to the server

[0212] The device uses the JavaScript fetch API to send the entered question to the server as an HTTP POST request. The body of the request contains the question entered by the user. The input is the question data entered by the user, and the output is the HTTP POST request sent to the server.

[0213] Step 4:

[0214] The server sends the question to the generative AI model

[0215] The server formats the received question as a prompt sentence and sends it to the generative AI model. The prompt sentence is sent to the generative AI model via an API request. The input is the question data received by the server, and the output is the prompt sentence sent to the generative AI model.

[0216] Step 5:

[0217] Generative AI models generate answers

[0218] The generative AI model generates an answer based on the prompt it receives. For example, in response to the question, "What is the best suggestion for this situation?", it generates an answer such as, "Customer, this product uses the latest technology and is very high-performance." The input is the prompt received by the generative AI model, and the output is the generated answer data.

[0219] Step 6:

[0220] The server receives the generated answer

[0221] The server receives the answer returned from the generative AI model in JSON format. The received JSON data contains the generated answer. The input is the answer data returned from the generative AI model, and the output is the JSON data received by the server.

[0222] Step 7:

[0223] The server sends the answer back to the device

[0224] The server returns the received answer to the terminal as an HTTP response. The body of the response contains the generated answer. The input is the JSON data received by the server, and the output is the HTTP response returned to the terminal.

[0225] Step 8:

[0226] The device displays the answer to the user

[0227] The terminal displays the received answer in a designated area on the interface. The user can check the displayed answer and decide the next action. The input is the HTTP response received by the terminal, and the output is the answer displayed on the interface.

[0228] (Application example 2)

[0229] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0230] Previous systems for improving store staff customer service skills had the problem that analyzing role-play videos and generating suggestion talks took time, making it difficult to provide real-time feedback. Furthermore, there was a lack of a way for store staff to instantly obtain the optimal suggestion talks on the spot, and effective support for improving the quality of customer service was not provided. This could delay the improvement of store staff customer service skills and lead to a decline in customer satisfaction.

[0231] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0232] In this invention, the server includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, and interactive interface means for store staff to ask questions to the generative AI using a mobile information terminal and obtain optimal suggestion talk in real time. This enables store staff to instantly obtain optimal suggestion talk on the spot, improving the quality of customer service.

[0233] "Store staff" refers to employees who work in stores and handle customer service, product explanations, sales promotion, and other tasks.

[0234] "Role-play video" is a video recording of store staff simulating how to deal with customers.

[0235] "Analysis" refers to the act of evaluating the content of the role-playing videos and measuring the customer service skills of store staff and the effectiveness of their proposal talks.

[0236] "Generative AI" is a system that uses artificial intelligence technology to generate new information and suggestions based on input data.

[0237] A "mobile information terminal" is a portable information processing device such as a smartphone or tablet.

[0238] An "interactive interface" is an operation screen or input means that allows users to interact with the system in a two-way manner.

[0239] "Proposal talk" refers to the way and content of the talk that store staff give to customers to explain products and promote sales.

[0240] "Feedback" refers to evaluations and advice provided by generative AI to store staff to improve their customer service skills.

[0241] A system for implementing this invention includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, and interactive interface means for store staff to ask questions to the generative AI using a mobile information terminal and obtain optimal suggestion talk in real time.

[0242] System configuration

[0243] 1. Hardware:

[0244] Mobile information terminal: A portable information processing device such as a smartphone or tablet.

[0245] Server: A high-performance computer for processing data and running generative AI.

[0246] 2. Software:

[0247] Python: A programming language used to implement the program.

[0248] Tkinter: A GUI library for building user interfaces that run on mobile devices.

[0249] OpenAI API: An API for using generative AI models (e.g., GPT-3 (registered trademark)).

[0250] Processing flow

[0251] 1. Role-play video analysis:

[0252] The server receives role-playing videos of store staff and uses a video analysis algorithm to evaluate their customer service skills and the effectiveness of their proposal talk.

[0253] The analysis results are stored in a database and used for subsequent processing.

[0254] 2. Generate and submit a question:

[0255] Store staff use mobile information terminals to input questions to the generative AI.

[0256] For example, enter a question such as, "What is the best phrase to use when introducing a new product?"

[0257] 3. Answer generation using generative AI:

[0258] The server sends questions to the generative AI model through the OpenAI API.

[0259] The generative AI model generates the optimal suggested dialogue based on the input question and sends it back to the server.

[0260] 4. Interactive interface display:

[0261] The server transmits the generated proposal talk to a mobile information terminal and displays it to the store staff through an interactive interface.

[0262] Store staff can check the optimal suggestion talk in real time and use it in actual customer service.

[0263] Specific examples

[0264] For example, if a store staff member asks, "What is the best phrase to use when introducing a new product?", the generative AI will provide an answer such as, "This new product uses the latest technology and is extremely high-performance compared to other products. In particular, it features a long battery life and an easy-to-use design."

[0265] Prompt Sentence Examples

[0266] What's the best thing to say when introducing a new product?

[0267] In this way, store staff can instantly receive the most appropriate proposals on-site, improving the quality of customer service.

[0268] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0269] Step 1:

[0270] Role-play video analysis

[0271] The server receives role-playing videos of store staff. The received videos are input into an analysis algorithm to evaluate customer service skills and the effectiveness of the proposed talk. The analysis results are stored in a database and used for subsequent processing. The input is the role-playing video, and the output is the analysis results. Specific operations include frame analysis of the video, voice recognition, and emotion analysis.

[0272] Step 2:

[0273] Generate and submit a question

[0274] The user (store staff) uses a mobile information terminal to input a question to the generative AI. For example, they can input a question such as, "What is the best phrase to use when introducing a new product?" The input is the user's question, and the output is a prompt sent to the generative AI. Specifically, the system accepts text input through the terminal's interface.

[0275] Step 3:

[0276] Answer generation using generative AI

[0277] The server sends a question to the generative AI model via the OpenAI API. The generative AI model generates the optimal suggested dialogue based on the input question and replies to the server. The input is a prompt, and the output is the generated suggested dialogue. Specific operations include sending an API request and receiving a response.

[0278] Step 4:

[0279] Interactive interface display

[0280] The server sends the generated suggested talk to the mobile information terminal and displays it to the user through an interactive interface. The user can check the optimal suggested talk in real time and use it in actual customer service. The input is the generated suggested talk, and the output is the text displayed on the user's terminal. Specific operations include converting the text format and displaying it on the interface.

[0281] Example 3

[0282] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] Conventional store staff proposal talks depended on their individual experience and skills, resulting in inconsistency and inconsistency in effectiveness. Furthermore, there was a lack of a feedback loop to generate effective proposal talks, making it difficult to improve staff skills.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0285] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, means for presenting the generated suggestion talk to the store staff, means for receiving feedback from the store staff, and means for retraining the generative AI model based on the received feedback. This enables store staff to consistently use effective suggestion talk and continuously improve their skills through feedback.

[0286] "Store staff" refers to employees who deal with customers and engage in sales activities in stores.

[0287] "Role-play video" refers to footage of store staff simulating how to deal with customers.

[0288] "Means of analysis" refers to the techniques and methods used to analyze role-play videos and evaluate the responses and suggestions of store staff.

[0289] "Means of asking questions" refers to the methods and techniques that store staff use to ask appropriate questions to customers.

[0290] "Generative AI methods" refer to technologies and methods for generating effective proposal talks using artificial intelligence.

[0291] "Proposal talk" refers to the content of proposals and explanations given by store staff to customers.

[0292] "Means for presentation" refers to the method or technology for displaying the generated proposal talk to store staff.

[0293] "Means for receiving feedback" refers to methods and techniques for collecting evaluations and opinions from store staff.

[0294] "Means for retraining" refers to techniques and methods for retraining a generative AI model based on received feedback to improve its accuracy.

[0295] MODE FOR CARRYING OUT THE INVENTION

[0296] This invention is a system that allows store staff to generate effective proposal talks and use them in customer service. The system operates mainly with a server, terminals, and users (store staff).

[0297] server

[0298] The server uses a generative AI model to generate appropriate suggestion speech in response to questions from store staff. This generation uses natural language generation technology. Specifically, it utilizes generative AI models such as OpenAI's GPT-4. The server performs the following processes:

[0299] 1. Receive a user question

[0300] The server receives a question from a user (store staff member). For example, the user inputs, "If a customer is looking for a new smartphone, what suggestions should I make?"

[0301] 2. Send a prompt to the generative AI model

[0302] The server sends the received question to the generative AI model as a prompt, which is then converted into a format that the generative AI model can easily understand.

[0303] 3. Receive suggested talks from the generative AI model

[0304] The server receives the proposed speech generated by the generative AI model, such as "This smartphone is equipped with the latest camera functions and is ideal for customers who love photography."

[0305] 4. Send the proposed chat to the device

[0306] The server then sends the generated proposal token to the terminal, which is a device used by the store staff.

[0307] 5. Retraining generative AI models based on feedback

[0308] The server retrains the generative AI model based on the feedback received from the user, allowing the model to generate more effective suggested conversations.

[0309] Terminal

[0310] The terminal receives the suggestion message sent from the server and displays it to the store staff. The terminal is equipped with an interactive interface, allowing the store staff to check the suggestion message and provide feedback as needed.

[0311] 1. Display suggested talks

[0312] The terminal displays the proposed talks sent from the server to the user, and allows the user to check the proposed talks through an interactive interface.

[0313] 2. Receive user feedback

[0314] The device receives feedback provided by the user in response to the proposed talk, such as "This suggestion was effective" or "I would like more specific information."

[0315] User (store staff)

[0316] The user checks the proposed conversations displayed on the device and uses them in actual customer service. In addition, the user provides data to the generative AI to learn from highly successful proposed conversations. The generative AI then uses that data to generate new proposed conversations.

[0317] 1. Check the proposal talk and use it in customer service

[0318] The user can check the suggested conversation displayed on the device and use it in actual customer service. For example, the user might suggest to a customer, "This smartphone is equipped with the latest camera functions, making it ideal for customers who like taking photos."

[0319] 2. Provide feedback

[0320] The user provides feedback on the effectiveness of the proposal talk, for example, by inputting specific feedback such as "This proposal was very well received by the customer."

[0321] Specific examples

[0322] Example prompts to be input to the generative AI model

[0323] "If a customer is looking for a new smartphone, what suggestions should you make?"

[0324] This system allows store staff to always use the latest and most effective suggestion talks, improving the quality of customer service. In addition, the generative AI model continuously learns through feedback, making it possible to improve the accuracy of suggestion talks. The flow of the identification process in Example 3 is explained using Figure 15.

[0325] Step 1:

[0326] The server receives a question from the user (store staff). The user types into the terminal, "If a customer is looking for a new smartphone, what suggestions should I make?" The server receives this question as a prompt. The input is the user's question, and the output is the prompt.

[0327] Step 2:

[0328] The server sends the received prompt to a generative AI model. Specifically, the prompt is input to the generative AI model (e.g., GPT-4) and data processing is performed to generate appropriate suggested speech. The input is the prompt, and the output is the input data for the generative AI model.

[0329] Step 3:

[0330] The server receives the proposed conversation generated by the generative AI model. The generative AI model generates the proposed conversation using natural language generation technology based on the prompt sentence. For example, it might generate something like, "This smartphone is equipped with the latest camera functions and is ideal for customers who love photography." The input is the input data to the generative AI model, and the output is the generated proposed conversation.

[0331] Step 4:

[0332] The server sends the generated suggestion token to the terminal. The terminal is a device used by store staff and has an interface for displaying the suggestion token. The input is the generated suggestion token, and the output is the data sent to the terminal.

[0333] Step 5:

[0334] The terminal receives the proposed tokens sent from the server and displays them to the user. The terminal allows the user to check the proposed tokens through an interactive interface. The input is the data sent from the server, and the output is the data displayed to the user.

[0335] Step 6:

[0336] The user checks the suggestion speech displayed on the device and uses it in actual customer service. For example, a user suggests to a customer, "This smartphone is equipped with the latest camera functions and is perfect for customers who like photography." The input is the suggestion speech displayed on the device, and the output is the suggestion to the customer.

[0337] Step 7:

[0338] The user provides feedback on the effectiveness of the proposal talk. For example, the user inputs specific feedback such as "This proposal was very well received by the customer" into the terminal. The input is information about the effectiveness of the proposal talk, and the output is feedback data.

[0339] Step 8:

[0340] The terminal receives feedback from the user and transmits it to the server. The input is the feedback data from the user, and the output is the data to be transmitted to the server.

[0341] Step 9:

[0342] The server retrains the generative AI model based on the received feedback, allowing the model to generate more effective suggested speeches. The input is the feedback data, and the output is the retrained generative AI model.

[0343] Step 10:

[0344] The server generates new suggested conversations using the retrained generative AI model. This allows the server to provide more accurate suggested conversations for subsequent user questions. The input is the retrained generative AI model, and the output is the newly generated suggested conversation.

[0345] (Application example 3)

[0346] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] Conventional store staff proposal conversations depended on their individual experience and skills, and the ability of the staff to make effective proposals was a major factor. Furthermore, there was a lack of real-time support for staff to make appropriate proposals to customers, and a lack of means to receive feedback on the proposal conversations. This led to issues such as insufficient improvement of customer satisfaction and sales promotion.

[0348] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0349] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective proposal talk in response to the questions, means for presenting the proposal talk generated by the generative AI means to the store staff in real time, and means for recording the proposal talk of the store staff and providing it as feedback later. This enables the store staff to make effective suggestions to customers in real time and also receive feedback to improve the quality of the proposal talk.

[0350] "Store staff" refers to employees who deal with customers and promote sales in physical stores.

[0351] "Role-play video" refers to footage of store staff simulating customer service interactions.

[0352] "Analysis" refers to the process of analyzing role-play videos and other data to extract specific patterns and trends.

[0353] A "question" is a question that a store staff member asks a customer to elicit information.

[0354] "Proposal talk" refers to explanatory and persuasive talk given by store staff to customers in order to recommend products or services.

[0355] "Generative AI" refers to a system that uses artificial intelligence technology to generate natural language and create sentences and conversations for specific purposes.

[0356] "Real-time" refers to a state in which data and information are processed immediately and provided without delay.

[0357] "Recording" refers to the process of saving store staff's suggestion talks and other data for future reference.

[0358] "Feedback" refers to the process of providing improvements and evaluations in response to suggestions made by store staff.

[0359] The system for implementing this invention is configured as follows: First, software for analyzing role-playing videos of store staff is installed on a server. This software uses video analysis technology to extract patterns of staff's proposal talk and customer interactions.

[0360] The server then uses natural language processing technology to generate questions for store staff to ask customers based on the analysis results. The generated questions are then sent to the staff's smartphones in real time.

[0361] Furthermore, the server uses a generative AI model to generate effective proposal speech in response to customer questions. This generative AI model has previously learned the proposal speech of highly successful store staff, and generates new proposal speech based on that knowledge. The generated proposal speech is also displayed in real time on the staff's smartphone.

[0362] The server also records the actual proposal talks made by the store staff and provides the records as feedback later. This feedback includes specific advice for improving the quality of the proposal talks made by the staff.

[0363] In implementing this system, the following hardware and software are used:

[0364] Hardware: Smartphones, servers

[0365] Software: Video analysis software, natural language processing software, generative AI models (e.g., OpenAI GPT)

[0366] As a concrete example, the following prompt sentence is used:

[0367] Learning prompt: Learn from this talk: This product is very popular, especially this time of year.

[0368] Proposal talk generation prompt: Generate a proposal talk for this query: I'd like to know more about this product.

[0369] Feedback prompt: Provide feedback for this talk: This product is very popular, especially this time of year.

[0370] This system allows store staff to make effective suggestions to customers in real time and receive feedback to improve the quality of their suggestions.

[0371] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0372] Step 1:

[0373] The server receives role-playing videos of store staff and analyzes them using video analysis software. The input is the role-playing video, and the output is the analysis results, including the staff's suggestion talk and customer response patterns. Specifically, the system analyzes the video frames and converts the talk content into text using voice recognition technology.

[0374] Step 2:

[0375] The server generates questions for store staff to ask customers based on the analysis results obtained in step 1. The input is the analysis results, and the output is the generated questions. Specifically, it uses natural language processing software to generate appropriate questions and sends them to the staff's smartphones.

[0376] Step 3:

[0377] The server uses a generative AI model to generate effective suggestion speech in response to customer questions. The input is the customer question, and the output is the generated suggestion speech. Specifically, the server inputs a prompt sentence into the generative AI model, and displays the generated speech on the staff member's smartphone in real time.

[0378] Step 4:

[0379] The terminal (smartphone) displays the proposed chat received from the server to the store staff. The input is the proposed chat sent from the server, and the output is the chat displayed on the staff's screen. Specifically, the proposed chat is displayed on the smartphone display so that the staff can check it.

[0380] Step 5:

[0381] The server records the actual proposal conversations made by store staff. The input is the staff's proposal conversations, and the output is the recorded conversation data. Specifically, the server uses voice recognition technology to convert the staff's conversations into text and saves it in a database.

[0382] Step 6:

[0383] The server generates feedback based on the recorded suggestion talk and provides it to the store staff. The input is the recorded talk data, and the output is the generated feedback. Specifically, the server inputs feedback prompts into the generative AI model and sends the generated feedback to the staff's smartphone.

[0384] Step 7:

[0385] The terminal (smartphone) displays the feedback received from the server to the store staff. The input is the feedback sent from the server, and the output is the feedback displayed on the staff's screen. Specifically, the feedback is displayed on the smartphone display so that the staff can check it.

[0386] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0387] "Example 1"

[0388] One embodiment of the present invention provides a system including a means for analyzing role-playing videos of store staff, a means for the store staff to ask questions based on the analysis results, a generative AI means for generating effective suggested talk in response to the questions, and an emotion engine for recognizing user emotions. Specifically, the store staff film role-playing videos simulating customer service scenes and input the videos into the generative AI. The generative AI analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions. Furthermore, the emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI. The generative AI optimizes the suggested talk to the store staff based on this feedback.

[0389] "Example 2"

[0390] As a specific example, consider the case where a store staff member films a role-play video of themselves handling a customer complaint and then inputs that video into a generative AI. The generative AI analyzes the words and attitude of the store staff member in the video, as well as the customer's reaction. Furthermore, the emotion engine analyzes the emotional state of the store staff member and feeds the results back to the generative AI. Based on this feedback, the generative AI optimizes the suggested dialogue to the store staff member. For example, if a store staff member is feeling irritated by a customer complaint, the generative AI will take that emotional state into consideration and generate suggested dialogue to respond more calmly.

[0391] "Example 3"

[0392] One embodiment of the present invention provides a system including a means for analyzing role-playing videos of store staff, a means for the store staff to ask questions based on the analysis results, a generative AI means for generating effective suggested talk in response to the questions, and an emotion engine for recognizing user emotions. Specifically, the store staff film role-playing videos simulating customer service scenes and input the videos into the generative AI. The generative AI analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions. Furthermore, the emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI. The generative AI optimizes the suggested talk to the store staff based on this feedback.

[0393] The processing flow of each embodiment will be described below.

[0394] "Example 1"

[0395] Step 1: Store staff film role-playing videos simulating customer service situations.

[0396] Step 2: Input the captured video into the generative AI.

[0397] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[0398] Step 4: The emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI.

[0399] Step 5: Based on this feedback, the generative AI optimizes the proposal talk to store staff.

[0400] "Example 2"

[0401] Step 1: Store staff film a role-play video of themselves handling a customer complaint. Step 2: The filmed video is input into the generative AI.

[0402] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[0403] Step 4: The emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI.

[0404] Step 5: The generative AI uses this feedback to optimize the suggested conversations to store staff. For example, if a store staff member is feeling irritated by a customer complaint, the generative AI will take that emotional state into account and generate a suggested conversation that will help them respond more calmly.

[0405] Example 1

[0406] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] Conventional training methods for improving store staff's customer service skills have made it difficult to effectively analyze staff words, attitudes, and customer reactions and provide feedback. Furthermore, they do not generate suggested conversations that take into account the emotional state of staff, making it difficult to respond effectively in actual customer service situations. This has led to the issue of insufficient improvement of staff customer service skills.

[0408] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0409] In this invention, the server includes means for shooting role-playing videos of store staff, means for uploading the shot videos to the server, means for analyzing the words, attitudes, and customer reactions in the videos, means for analyzing the emotional state of the store staff, generative AI means for generating suggestive talk based on the results of the emotional analysis, and means for checking the generated suggestive talk and utilizing it in actual customer service, thereby making it possible to effectively improve the customer service skills of store staff.

[0410] "Store staff" refers to employees who serve customers in a store.

[0411] "Role-play videos" are videos filmed by store staff simulating customer service situations.

[0412] A "server" is a computer system for storing, processing, and analyzing data.

[0413] "Upload" is the act of a user sending data from a local device to a server.

[0414] "Generative AI" is a system that uses artificial intelligence technology to analyze data and generate output tailored to specific purposes.

[0415] "Emotional state" refers to changes in the psychological state and emotions of store staff.

[0416] "Proposal talk" refers to the content of proposals and explanations that store staff give to customers during customer service situations.

[0417] "Feedback" is the act of providing improvements and advice based on analysis results and evaluations.

[0418] "Analysis" is the process of examining data in detail to understand its meaning and patterns.

[0419] A "specialized application" is a software program designed for a specific purpose.

[0420] This invention provides a system for improving the customer service skills of store staff, specifically, a technology for analyzing role-play videos and generating suggestive talks. An embodiment of this system will be described in detail below.

[0421] First, the user (store staff) uses a camera such as a smartphone or video camera to shoot a role-play video simulating a customer interaction scene. Once the shooting is complete, the video file is saved.

[0422] Next, the user launches the dedicated application and uploads the saved role-play video to the server. The dedicated application is video uploading software designed to allow users to easily send videos to the server.

[0423] The server receives the uploaded video file and uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the words, attitudes, and customer reactions in the video. The generative AI model converts the audio data in the video into text and performs a detailed analysis of the words, attitudes, and customer reactions.

[0424] Furthermore, the server uses an emotion recognition engine (e.g., Affectiva) to analyze the emotional state of the store staff. The emotion recognition engine analyzes the facial expressions and tone of voice of the store staff in the video and generates numerical data representing their emotional state.

[0425] The server feeds back the results of the emotion analysis to the generative AI to generate the optimal suggestion speech. The generative AI then generates a prompt based on the feedback and creates a suggestion speech for the store staff.

[0426] Users can check the generated suggested conversations through a dedicated application and use them in actual customer service situations, allowing store staff to acquire more effective customer service skills.

[0427] As a concrete example, the following prompt sentence can be input to a generative AI model:

[0428] Example prompt sentence:

[0429] "If a store staff member shows signs of dissatisfaction while serving a customer, please suggest how they should respond."

[0430] This system will enable store staff to improve their ability to respond to customers in actual customer service situations, and is expected to contribute to improving customer satisfaction.

[0431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] The user uses a smartphone or video camera to shoot a role-play video simulating a customer service scene.

[0434] Input: Mock customer service scene

[0435] Output: Roleplay video file

[0436] Specific operation: The user launches the camera app on their smartphone and starts a role-play simulating a customer service scene. Once the recording is complete, the video file is saved.

[0437] Step 2:

[0438] The user launches the dedicated application and uploads the saved role-play video to the server.

[0439] Input: Roleplay video file

[0440] Output: Video data stored on the server

[0441] Specific operation: The user launches the dedicated application, selects the saved video file, and presses the upload button. The video is then sent to the server.

[0442] Step 3:

[0443] The server uses a generative AI model to analyze the words, attitudes, and customer reactions in the video.

[0444] Input: Video data stored on the server

[0445] Output: Analysis results (words, attitudes, and customer response data)

[0446] How it works: The server receives the uploaded video file and inputs it into a generative AI model. The AI ​​model converts the audio data in the video into text and analyzes the words, attitude, and customer reactions.

[0447] Step 4:

[0448] The server uses an emotion recognition engine to analyze the emotional state of the store staff.

[0449] Input: Analysis results (words, attitudes, customer response data)

[0450] Output: Emotional state data

[0451] Specific operation: The server inputs the facial expressions and tone of voice of the store staff in the video into an emotion recognition engine to analyze their emotional state. The analysis results are generated as numerical data.

[0452] Step 5:

[0453] The server feeds back the results of the emotion analysis to the generative AI to generate the optimal suggested talk.

[0454] Input: Emotional state data

[0455] Output: Proposal Talk

[0456] How it works: The server sends the results of the emotion analysis to the generative AI, which then generates prompts based on the feedback and creates the optimal suggested conversation.

[0457] Step 6:

[0458] Users can check the generated suggestion talk through a dedicated application and use it in actual customer service.

[0459] Input: Suggested Talk

[0460] Output: Improved customer service skills

[0461] Specific operation: The user checks the generated suggestion talk through a dedicated application, reads the suggestion talk, and uses it in actual customer service situations.

[0462] (Application example 1)

[0463] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0464] Conventional training methods for improving store staff's customer service skills have made it difficult to effectively analyze staff language, attitudes, and customer reactions, and provide optimal suggestions. Furthermore, feedback that does not take into account the staff's emotional state is difficult to provide, making it difficult for them to respond effectively in actual customer service situations. This has limited the improvement of staff's customer service skills, and has also limited the improvement of customer satisfaction.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0466] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state of the store staff, means for feeding back the results of the emotion engine means to the generative AI means, and means for the generative AI means to generate optimal suggestion talk based on the feedback. This makes it possible to effectively analyze the language and attitude of staff and the reactions of customers, and to provide optimal suggestion talk that takes into account their emotional state.

[0467] "Store staff" refers to employees who provide customer service to customers in physical stores.

[0468] "Role-play videos" are videos filmed by store staff simulating customer service situations.

[0469] "Means of analysis" refers to technology and equipment used to analyze the words and attitudes of store staff and customer reactions in role-play videos.

[0470] "Means for asking questions" refers to the technology or devices that allow store staff to ask appropriate questions to customers based on the analysis results.

[0471] "Generative AI means" refers to technologies and devices that use artificial intelligence technology to generate effective suggestion talk in response to questions.

[0472] The "emotion engine means" refers to a technology or device for analyzing the emotional state of store staff.

[0473] "Feedback means" refers to the technology or device used to transmit the results of the emotion engine means to the generative AI means.

[0474] "Means for generating optimal proposal talk" refers to the technology or device that enables the generative AI means to generate optimal proposal talk for store staff based on feedback.

[0475] A system for implementing this invention analyzes role-play videos and generates optimal proposal talks to improve the customer service skills of store staff. A specific embodiment of this system will be described below.

[0476] System Configuration

[0477] The system consists of the following major components:

[0478] 1. Video recording device: Store staff will record role-play videos using a device with a camera function, such as a smartphone.

[0479] 2. Server: A central processing unit that performs video analysis, sentiment analysis, and generates suggested talks using generative AI.

[0480] 3. Emotion Engine: Software for analyzing the emotional state of store staff.

[0481] 4. Generative AI model: An artificial intelligence model that generates optimal suggested talks based on the results of video analysis and sentiment analysis.

[0482] Video recording and uploading

[0483] Store staff use their smartphones to film their customer service role-playing videos, which are then uploaded to a server via a dedicated application.

[0484] Video Analysis

[0485] The server receives the uploaded video and divides it into frames using a video analysis library such as OpenCV. Each frame is analyzed to determine the language and attitude of store staff and the customer's reaction.

[0486] sentiment analysis

[0487] The server uses an emotion engine to analyze the emotional state of the store staff in each frame. The emotion engine analyzes facial expressions, tone of voice, etc., and quantifies the emotional state of the staff.

[0488] Proposal talk generation

[0489] The server uses a generative AI model to combine the results of video analysis and sentiment analysis to generate optimal suggested talks. The generative AI model learns from past successful suggested talks and provides feedback to staff based on this.

[0490] Providing feedback

[0491] The generated suggestion talk and feedback are provided to store staff via a smartphone application, allowing them to improve their customer service skills.

[0492] Specific examples

[0493] For example, a store staff member uploads a video following the prompt: "Please upload a customer service role-playing video. We will analyze the language, attitude, and customer reactions in the video and provide the optimal suggestion talk." The server analyzes the video and evaluates the staff member's language, attitude, and customer reactions. The emotion engine analyzes the staff member's emotional state and provides feedback to the generative AI. The generative AI then generates the optimal suggestion talk and provides it to the staff member.

[0494] In this way, store staff will be able to respond effectively in actual customer service situations.

[0495] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0496] Step 1:

[0497] The user uses a smartphone to film a customer service role-play video.

[0498] Input: Customer Service Roleplay Video

[0499] Output: Recorded video file

[0500] Specific operation: The user starts the camera app on their smartphone and records a role-play simulating a customer service scene. Once the recording is complete, the video file is saved on the smartphone.

[0501] Step 2:

[0502] Users upload videos they have taken to the server using a dedicated application.

[0503] Input: Recorded video file

[0504] Output: Video file uploaded to the server

[0505] Specific operation: The user launches the dedicated application, selects the video file they have taken, and presses the upload button. The application then sends the video file to the server.

[0506] Step 3:

[0507] The server receives the uploaded video and splits it into frames using OpenCV.

[0508] Input: Uploaded video file

[0509] Output: A list of split frame images

[0510] Specific operation: The server receives a video file and splits it into frames using the OpenCV library. Each frame image is saved as a list.

[0511] Step 4:

[0512] The server uses a generative AI model to analyze the language and attitude of store staff and customer reactions in each frame.

[0513] Input: A list of split frame images

[0514] Output: Analysis of language, attitude, and customer response

[0515] How it works: The server uses a generative AI model to analyze each frame image. As a result, the store staff's language and attitude, as well as the customer's reaction, are quantified.

[0516] Step 5:

[0517] The server uses an emotion engine to analyze the emotional state of the store staff in each frame.

[0518] Input: A list of split frame images

[0519] Output: Analysis of store staff's emotional state

[0520] Specific operation: The server uses the emotion engine to analyze the emotional state of the store staff in each frame image. The emotional state is quantified and saved as the analysis result.

[0521] Step 6:

[0522] The server uses a generative AI model to integrate the results of video analysis and sentiment analysis to generate the optimal suggested talk.

[0523] Input: Analysis of language, attitude, customer reactions, and store staff emotional state

[0524] Output: Best suggestion talk

[0525] How it works: The server uses a generative AI model to integrate the results of video analysis and sentiment analysis. Based on the integrated results, the optimal suggested conversation is generated.

[0526] Step 7:

[0527] The server provides the generated proposed talk and feedback to the user through a smartphone application.

[0528] Input: Best Suggestion Talk

[0529] Output: Suggested talk and feedback provided to the user

[0530] Specific operation: The server sends the generated proposed talk and feedback to a dedicated application, and the user receives the proposed talk and feedback through the application.

[0531] The above are the specific processing steps for carrying out the present invention.

[0532] Example 2

[0533] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0534] Conventional store staff training systems lack specific feedback to improve staff response skills. Furthermore, it is difficult to generate suggestive conversations that take into account the emotional state of staff, making it difficult to effectively handle complaints. This can delay the improvement of staff response skills and lead to a decline in customer satisfaction.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0536] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state, and means for returning the generated suggestion talk to the store staff. This enables specific feedback to improve the response capabilities of the store staff, and realizes the generation of effective suggestion talk that takes the emotional state into consideration.

[0537] "Store staff" refers to employees who deal with customers and perform sales duties in a store.

[0538] A "role-play video" is a video in which store staff act out specific situations and are filmed.

[0539] "Means of analysis" refers to the technology and equipment used to analyze the content of role-play videos and evaluate the behavior and words of store staff and the reactions of customers.

[0540] The "means of asking questions" refers to an interface or system that allows store staff to input questions into the generative AI and obtain information based on those questions.

[0541] "Generative AI means" refers to technologies or devices that use artificial intelligence technology to generate optimal proposal talks based on input data.

[0542] "Emotion engine means" refers to technology or equipment that analyzes the emotional state of people in a video and feeds the results back to the generative AI.

[0543] "Suggestion talk" refers to the optimal response or comments that store staff should make to customers in a specific situation.

[0544] "Means of providing feedback" refers to the technology and devices used to communicate the proposed talk and analysis results generated by generative AI to store staff.

[0545] This invention is a system for improving the customer service skills of store staff, specifically, analyzing role-play videos and generating optimal proposal talks. A specific embodiment of this system will be described below.

[0546] First, the user (store staff) uses a device such as a smartphone or tablet to film a role-play video of handling a complaint. This video records the store staff acting out a specific situation.

[0547] Next, the user accesses the server through a web browser or a dedicated application. The interface includes a video upload function and a question input field. The user uploads the role-play video they have filmed to the server through the interface.

[0548] The server receives the uploaded video and inputs it into a generative AI model. This generative AI model uses OpenAI's GPT-4 or Google Cloud Vision API, for example, to analyze the words and attitudes of store staff in the video, as well as customer reactions. Specifically, it uses natural language processing (NLP) technology to analyze the content of conversations, and image recognition technology to analyze facial expressions and gestures.

[0549] Furthermore, the server uses an emotion engine to analyze the emotional state of the store staff in the video. The emotion engine can use, for example, the Emotion API from Microsoft® Azure®. The analysis results of the emotion engine are fed back to the generative AI model.

[0550] The generative AI model uses this feedback to generate optimal suggested dialogue. For example, if a store staff member appears frustrated in the video, the generative AI model takes that emotion into account and generates specific suggested dialogue for responding calmly.

[0551] Finally, the server returns the generated proposed token to the user, who can then check it through the interface and use it in their work.

[0552] As a concrete example, consider the case where a user inputs the following prompt sentence into a generative AI model:

[0553] Example prompt sentence:

[0554] "We have filmed a role-play video of us handling a customer complaint. Please analyze this video and tell us the best way to present our proposal."

[0555] In this way, users can receive suggestions from the generative AI and use them in their work. This system enables specific feedback to improve the responsiveness of store staff, and generates effective suggestion conversations that take into account their emotional state.

[0556] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0557] Step 1:

[0558] The user shoots a role-play video.

[0559] Input: A situation in which a user takes a photo using a device such as a smartphone or tablet.

[0560] Specific operation: The user acts out a situation in which they are handling a complaint and takes a picture of the situation using the device's camera.

[0561] Output: The captured role-play video file.

[0562] Step 2:

[0563] A user uploads a video to a server.

[0564] Input: Filmed roleplay video file.

[0565] Specific operation: Users access the interface through a web browser or dedicated application, select a video file, and upload it.

[0566] Output: Roleplay video files stored on the server.

[0567] Step 3:

[0568] The server inputs the video into a generative AI model.

[0569] Input: Roleplay video files stored on the server.

[0570] Specific operation: The server converts the video file into the appropriate format and inputs it into the generative AI model.

[0571] Output: Video data fed into the generative AI model.

[0572] Step 4:

[0573] A generative AI model analyzes the video.

[0574] Input: Video data fed into the generative AI model.

[0575] How it works: The generative AI model uses natural language processing (NLP) techniques to analyze conversational content and image recognition techniques to analyze facial expressions and gestures, such as OpenAI's GPT-4 and Google Cloud Vision API.

[0576] Output: Analysis results of the words and attitudes of store staff in the video and customer reactions.

[0577] Step 5:

[0578] The emotion engine analyzes the emotional state.

[0579] Input: Analysis results of the words and attitudes of store staff in the video and customer reactions.

[0580] How it works: The emotion engine uses, for example, Microsoft Azure's Emotion API to identify emotions from the facial expressions and tone of voice of store staff.

[0581] Output: Analysis of store staff's emotional state.

[0582] Step 6:

[0583] A generative AI model generates suggested talks.

[0584] Input: Analysis results on the emotional state of store staff.

[0585] Specific behavior: The generative AI model generates optimal suggestion speech based on feedback from the emotion engine. For example, if a store staff member is feeling irritated, it takes that emotion into account and generates specific suggestion speech to respond calmly.

[0586] Output: The generated proposal talk.

[0587] Step 7:

[0588] The server returns the generated suggested token to the user.

[0589] Input: The generated proposal token.

[0590] Specific operation: The server receives the proposed talk generated from the generative AI model and returns it to the user through the interface.

[0591] Output: A suggested talk that the user can see.

[0592] In this way, users can receive suggestions from generative AI and use them in their actual work.

[0593] (Application example 2)

[0594] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0595] In the past, it was difficult for store staff to generate optimal suggestions that took into account the emotional state of the staff and the customer's reaction when handling complaints. Furthermore, there was a lack of means for staff to reflect on their own responses and receive concrete feedback to improve. This could delay the improvement of staff's response skills and lead to a decline in customer satisfaction.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0597] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state, and means for feeding back the results of the emotion engine means to the generative AI means. This makes it possible to generate optimal suggestion talk that takes into account the emotional state of the store staff and the reactions of customers, which is expected to improve the staff's response skills and customer satisfaction.

[0598] "Store staff" refers to employees who deal with customers and perform sales duties in physical stores.

[0599] "Role-play video" refers to footage of store staff simulating how to deal with customers.

[0600] "Means of analysis" refers to the technology and equipment used to analyze the content of role-play videos and evaluate the behavior and words of store staff and the reactions of customers.

[0601] The "means of asking questions" refers to the interface or method that store staff use to input specific questions to the generative AI.

[0602] "Generative AI means" refers to artificial intelligence that generates optimal proposal talks based on input data.

[0603] "Emotion engine means" refers to technology or equipment that analyzes the emotional state of store staff in videos and feeds the results back to the generative AI.

[0604] "Feedback means" refers to the methods and technologies for transmitting the analysis results of the emotion engine means to the generative AI means.

[0605] "Suggestion talk" refers to the optimal words and phrases that store staff can use when dealing with customers.

[0606] A system for implementing this invention includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing emotional states, and means for feeding back the results of the emotion engine means to the generative AI means.

[0607] System configuration

[0608] The system consists of the following main components:

[0609] 1. Smartphone: A device used by store staff to record role-play videos of customer complaint handling and to run applications.

[0610] 2. Server: A central processing unit that analyzes videos, analyzes emotional states, and generates suggested talks using generative AI.

[0611] 3. Generative AI model: An artificial intelligence model for generating proposed conversations. Specifically, we use a natural language generation model such as GPT-2.

[0612] 4. Emotion Engine: A model for analyzing the emotional state of store staff in videos. Specifically, we use an emotion analysis model using TENSORFLOW (registered trademark).

[0613] Video analysis and emotional state analysis

[0614] The server receives role-play videos uploaded from smartphones. The videos are analyzed frame by frame, and the store staff's behavior and customer reactions are evaluated. The emotion engine preprocesses each frame in the video and predicts the store staff's emotional state. This makes it clear what emotional state the staff is in when responding.

[0615] Proposal talk generation

[0616] The generative AI model receives feedback from the emotion engine and generates optimal suggestions that take into account the emotional state of the store staff and the customer's reaction. For example, if a store staff member is in an irritated emotional state and the customer is reacting angrily, the generative AI model will generate suggestions such as "understand the customer's feelings and offer an apology."

[0617] Specific examples

[0618] As a specific use case, consider the case where a store staff member asks, "If a customer is angry, what is a suggested conversation to respond calmly?" A role-play video taken with a smartphone is uploaded to the server, and the emotion engine analyzes the staff member's emotional state. The generative AI model generates a suggested conversation based on the following prompt:

[0619] If a store staff member is in an irritated emotional state and a customer is reacting angrily, what is the best suggestion talk?

[0620] In this way, store staff can reflect on their own responses and identify specific areas for improvement, which is expected to improve staff response skills and customer satisfaction.

[0621] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0622] Step 1:

[0623] The user (store staff) uses a smartphone to shoot a role-play video of handling a complaint. The video is saved on the smartphone.

[0624] Step 2:

[0625] The user starts the application on their smartphone and uploads the role-playing video they have taken to the server. The input is the role-playing video, and the output is the video data sent to the server.

[0626] Step 3:

[0627] The server analyzes the received role-play video. Specifically, it divides the video into frames and preprocesses the image data of each frame. The input is the video data, and the output is the preprocessed frame data.

[0628] Step 4:

[0629] The server inputs the preprocessed frame data into the emotion engine to analyze the emotional state of the store staff. The emotion engine predicts the emotional state from the image data of each frame and generates a result. The input is the preprocessed frame data, and the output is the predicted emotional state.

[0630] Step 5:

[0631] The server feeds back the emotional state prediction results obtained from the emotion engine to the generative AI model. The input is the emotional state prediction results, and the output is feedback data to the generative AI model.

[0632] Step 6:

[0633] A user inputs a question to the generative AI model through a smartphone application. For example, they input a question like, "If a customer is angry, what is a suggested conversation to respond calmly?" The input is the user's question, and the output is prompt data for the generative AI model.

[0634] Step 7:

[0635] The generative AI model generates optimal suggested speech based on the predicted emotional state and the user's question. The input is the predicted emotional state and prompt data, and the output is the suggested speech.

[0636] Step 8:

[0637] The server sends the generated proposed token to the smartphone application. The input is the proposed token, and the output is the data to be sent to the smartphone.

[0638] Step 9:

[0639] The user can then review the generated suggested speech through a smartphone application, allowing them to reflect on their own actions and identify specific areas for improvement. The input is the suggested speech, and the output is the user's understanding and feedback.

[0640] Example 3

[0641] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0642] Previous systems for improving store staff customer service skills had the problem of being difficult to generate suggestive conversations and provide feedback that fully considered the skills and emotional state of each staff member. They also lacked a means to effectively convey the know-how of highly successful staff members to other staff members. This resulted in inconsistencies in the quality of customer service, making it difficult to improve overall customer satisfaction.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0644] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggested talk in response to the questions, means for presenting the generated suggested talk to the store staff, emotion engine means for recognizing the emotional state of the store staff, means for feeding back the results of the emotion engine means to the generative AI means, and means for providing feedback to the store staff based on the suggested talk generated by the generative AI means. This makes it possible to generate and provide feedback on optimal suggested talk that takes into account the individual skills and emotional states of store staff, thereby improving the quality of customer service and customer satisfaction.

[0645] "Store staff" refers to employees who handle customer service and product explanations in stores.

[0646] "Role-play videos" are videos filmed by store staff simulating customer service scenes.

[0647] The "analysis method" refers to the technology used to analyze the content of role-play videos and evaluate the behavior of store staff and customer reactions.

[0648] The "means for asking questions" refers to a technique that enables store staff to generate appropriate questions based on the analysis results and ask them to customers.

[0649] "Generative AI methods" are technologies that use artificial intelligence to generate effective proposals in response to questions from store staff.

[0650] "Proposal talk" refers to the product explanations and proposals made by store staff to customers.

[0651] The "emotion engine means" is a technology for recognizing the emotional state of store staff and analyzing that information.

[0652] The "means of providing feedback" is a technology that provides specific advice and guidance to store staff based on the proposed talk generated by the generative AI means.

[0653] "Highly performing store staff" are store staff who demonstrate excellent results in evaluation criteria such as customer satisfaction and sales performance.

[0654] The "learning means" is a technology that allows the generative AI means to learn the proposal talk of store staff who are achieving high results.

[0655] MODE FOR CARRYING OUT THE INVENTION

[0656] This invention is a system for improving the customer service skills of store staff, using a generative AI model and an emotion engine to generate effective suggestion talk and provide feedback.

[0657] System configuration

[0658] This system consists of three main components: a server, a terminal, and a user.

[0659] server

[0660] The server is equipped with a generative AI model and an emotion engine. The generative AI model uses natural language generation technology to generate appropriate suggestion speech in response to store staff questions. Specifically, the server uses machine learning technology to learn the suggestion speech of highly successful store staff and uses the results to generate new suggestion speech. In addition, the emotion engine recognizes the emotional state of store staff and feeds the results back to the generative AI model.

[0661] Terminal

[0662] The terminal is a device used by store staff, and displays the proposed talk generated by the generative AI through an interactive interface. In addition, the terminal also records role-playing videos of the store staff and sends the videos to a server.

[0663] User

[0664] The user (store staff) films a role-playing video simulating a customer service situation and sends it to the server via their device. The server analyzes the words, attitudes, and customer reactions in the video, and uses an emotion engine to recognize the store staff's emotional state. The results are fed back to the generative AI, which generates an optimized proposal conversation.

[0665] Hardware and software used

[0666] Hardware: Servers, smartphones, tablets

[0667] Software: Generative AI models (e.g., GPT-4), emotion engines, video analysis software (e.g., OpenCV), speech recognition software (e.g., Google Speech-to-Text)

[0668] Specific examples

[0669] As a concrete example, consider a scene in which a store staff member is introducing a new product. The user films the introduction of the new product as a role-play video and sends it to the server via their device. The server analyzes the video, and the generative AI generates an appropriate suggestion dialogue. For example, the generated suggestion dialogue might be, "This new product is especially popular with the younger generation. Would you like to try it?"

[0670] Prompt Sentence Examples

[0671] "Please film a role-play video introducing a new product and send it to the server. The generative AI will generate an appropriate proposal speech."

[0672] This system enables store staff to serve customers more effectively, and is expected to improve customer satisfaction. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0673] Step 1:

[0674] The user shoots a role-play video.

[0675] Specific operation: The user uses a smartphone or tablet to record role-play videos that simulate customer service scenes, such as introducing a new product or interacting with a customer.

[0676] Input: Customer service scene demonstration

[0677] Output: Roleplay video file

[0678] Step 2:

[0679] The video captured by the device is sent to the server.

[0680] Specific operation: Open the application on your device and tap the "Upload video" button. Select the video you have taken and send it to the server.

[0681] Input: Roleplay video file

[0682] Output: Video data uploaded to the server

[0683] Step 3:

[0684] The server analyzes the video.

[0685] Specific operation: The server analyzes the video using video analysis software (e.g., OpenCV), and converts the audio data into text using speech recognition software (e.g., Google Speech-to-Text).

[0686] Input: Uploaded video data

[0687] Output: Text data and analysis results

[0688] Step 4:

[0689] The server generates proposed talks using generative AI.

[0690] Specific operation: The server generates a suggested conversation using a generative AI model (e.g., GPT-4). The generated text is saved as a suggested conversation.

[0691] Input: Text data and analysis results

[0692] Output: Text data of the proposed talk

[0693] Step 5:

[0694] The server sends the generated proposed token to the terminal.

[0695] Specific operation: The server sends the generated proposed token to the device application, and the sent data is notified to the user through the device's notification function.

[0696] Input: Text data of the proposed talk

[0697] Output: Proposal token sent to the device

[0698] Step 6:

[0699] The terminal displays the suggested talk to the user.

[0700] Specific operation: Open the application on your device and check the suggested talk. The suggested talk will be displayed on the main screen of the application.

[0701] Input: Suggested token sent to the device

[0702] Output: Suggested speech displayed on the screen

[0703] Step 7:

[0704] The user provides customer service based on the proposed talk.

[0705] Specific operation: The user provides actual customer service based on the displayed suggestion talk. For example, the user introduces a product to a customer using the generated suggestion talk.

[0706] Input: Suggested speech displayed on the screen

[0707] Output: Actual customer service scene

[0708] Step 8:

[0709] The server learns the customer service results and updates the generative AI.

[0710] Specific operation: The server collects the results of customer service interactions and trains the generative AI. Specifically, it collects the success rate of interactions and customer reaction data and updates the generative AI model.

[0711] Input: Customer service result data

[0712] Output: An updated generative AI model

[0713] (Application example 3)

[0714] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0715] Conventional training methods for improving store staff's customer service skills make it difficult to properly evaluate the emotional state of individual staff members and the quality of their specific proposal talks, and provide feedback. Furthermore, there is a lack of specific means for generating effective proposal talks, which means it takes time to improve staff skills. This has led to the issue of delays in improving the quality of customer service across the entire store.

[0716] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0717] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state of the store staff, and means for providing feedback based on the generated suggestion talk and the emotional state, thereby making it possible to efficiently improve the customer service skills of store staff.

[0718] "Store staff" refers to employees who serve customers and make sales in physical stores.

[0719] "Role-play videos" refer to videos filmed by store staff simulating customer service situations.

[0720] "Means of analysis" refers to the technology and equipment used to analyze role-play videos and evaluate the words and attitudes of store staff, customer reactions, etc.

[0721] "Generative AI methods" refer to systems and programs that use artificial intelligence technology to generate effective suggestion talk in response to questions from store staff.

[0722] "Emotion engine means" refers to technology or devices that analyze the emotional state of store staff and provide feedback on the results to generative AI.

[0723] "Means for providing feedback" refers to a system or program for providing specific advice or guidance to store staff based on the generated suggestion talk and emotional state.

[0724] A system for carrying out this invention has the following configuration: A server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state of the store staff, and means for providing feedback based on the generated suggestion talk and emotional state.

[0725] Hardware and software used

[0726] Hardware: Smartphone (with camera)

[0727] Software: OpenCV (video processing), Transformers (GPT-3 model), EmotionRecognizer (emotion recognition)

[0728] Processing flow

[0729] 1. Loading a roleplay video:

[0730] Role-playing videos taken with a smartphone are uploaded to the server, which uses OpenCV to split the videos into frames and prepare them for analysis.

[0731] 2. Video Analysis:

[0732] The server analyzes the words and attitudes of store staff and customer reactions in the video, specifically using facial and voice recognition to extract the staff's facial expressions and what they say.

[0733] 3. Proposal talk generation:

[0734] The server uses the GPT-3 model to generate effective suggestions based on a customer service scenario. For example, it uses the following prompt sentence as input:

[0735] "Hello, customer. What product are you looking for today?"

[0736] 4. Sentiment Analysis:

[0737] The server uses EmotionRecognizer to analyze the emotional state of the store staff in the video, thereby understanding the emotional state of the staff when serving customers.

[0738] 5. Providing Feedback:

[0739] The server then provides specific advice and guidance to store staff based on the generated suggested speech and their emotional state, such as advice on how to improve the suggested speech or how to serve customers according to their emotional state.

[0740] Specific examples

[0741] For example, when a store staff member is serving a customer based on the prompt "Hello, customer. What kind of product are you looking for today?", the server generates the following suggestion dialogue:

[0742] "Hello, customer. What products would you like to purchase today? If you're not sure what you're looking for, let me introduce you to some new products, especially for this season."

[0743] Furthermore, if the emotion engine analyzes a staff member's emotional state as "tense," the server will provide feedback such as "Try to speak in a relaxed manner."

[0744] In this way, store staff can develop more effective customer service skills.

[0745] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0746] Step 1:

[0747] The server receives role-playing videos taken with a smartphone and splits the videos into frames using OpenCV. The input is a role-playing video file, and the output is a list of frames. Specifically, the server reads the video file, extracts each frame sequentially, and stores them in a list.

[0748] Step 2:

[0749] The server performs facial and voice recognition for each frame to extract the facial expressions and speech of the store staff. The input is a list of frames, and the output is a list of facial and voice recognition results. Specifically, it uses a facial recognition algorithm to detect faces in each frame, and a voice recognition engine to convert the speech into text.

[0750] Step 3:

[0751] The server uses the GPT-3 model to generate effective suggestion speech based on the customer service scenario. The input is the prompt sentence and the speech recognition result, and the output is the generated suggestion speech. Specifically, the prompt sentence is input into the GPT-3 model, and the generated text is obtained as the suggestion speech.

[0752] Step 4:

[0753] The server uses EmotionRecognizer to analyze the emotional states of store staff in the video. The input is a list of face recognition results, and the output is a list of emotional states. Specifically, the face images in each frame are input to EmotionRecognizer, and the emotional state is estimated.

[0754] Step 5:

[0755] The server provides specific advice and guidance to store staff based on the generated suggested speech and emotional state. The input is a list of suggested speech and emotional state, and the output is a feedback message. Specifically, the server evaluates the content of the suggested speech and the emotional state, and generates a feedback message containing suggestions for improvement and advice.

[0756] Step 6:

[0757] The server sends the generated feedback message to the store staff's smartphone. The input is the feedback message, and the output is a notification to the staff's smartphone. Specifically, the server uses a message sending API to send the feedback message to the smartphone.

[0758] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0759] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0760] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0761] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0762] [Second embodiment]

[0763] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0764] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0765] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0766] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0767] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0768] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0769] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0770] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0771] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0772] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0773] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0774] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0775] "Example 1"

[0776] This embodiment of the present invention uses video analysis technology as a means of analyzing role-playing videos of store staff. Specifically, store staff film role-playing videos simulating actual customer service scenes, and input the videos into a generative AI. The generative AI then analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions.

[0777] "Example 2"

[0778] Next, an interactive interface is provided as a means for store staff to ask questions to the generative AI. Specifically, store staff can ask the generative AI questions such as, "What is the best suggestion to make in this situation?"

[0779] "Example 3"

[0780] The generative AI then uses natural language generation technology to generate effective suggestion speeches. Specifically, the generative AI generates appropriate suggestion speeches in response to questions from store staff and presents them to them.

[0781] "Example 4"

[0782] Furthermore, machine learning technology is used to train the generative AI to learn the proposal speech of highly successful store staff. Specifically, the generative AI learns the proposal speech of highly successful store staff, and the results are used to generate new proposal speech.

[0783] "Example 5"

[0784] Finally, the interactive interface is used again to provide feedback to store staff based on the proposed speech learned by the generative AI. Specifically, the system provides specific advice and guidance to store staff based on the proposed speech generated by the generative AI.

[0785] The processing flow of each embodiment will be described below.

[0786] "Example 1"

[0787] Step 1: Store staff film role-playing videos simulating actual customer service situations.

[0788] Step 2: Input the filmed role-play video into the generative AI.

[0789] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[0790] "Example 2"

[0791] Step 1: Store staff ask questions to the generative AI through an interactive interface, such as, "What is the best suggestion to make in this situation?"

[0792] "Example 3"

[0793] Step 1: The generative AI generates an appropriate suggestion speech in response to the store staff's question. Step 2: The generated suggestion speech is presented to the store staff.

[0794] "Example 4"

[0795] Step 1: Have the generative AI learn the proposal talk of highly successful store staff.

[0796] Step 2: The generative AI uses the learning results to generate new proposed talks.

[0797] "Example 5"

[0798] Step 1: Based on the proposed conversation generated by the generative AI, specific advice and guidance is provided to store staff.

[0799] Example 1

[0800] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] Conventional training methods for improving store staff's customer service skills make it difficult to accurately evaluate the staff's actual customer service attitude and customer reactions, making it impossible to provide effective feedback.In addition, there was a lack of concrete measures to improve the quality of staff's suggestion talk, which limited the improvement in customer service skills.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0803] In this invention, the server includes means for shooting role-playing videos of store staff, means for uploading the shot videos to the server, means for receiving the uploaded videos and converting the speech into text using speech recognition technology, means for analyzing the text data and video data in the videos using a generative AI model, means for evaluating the customer service skills of the store staff based on the analysis results and generating feedback, and means for displaying the generated feedback to the user. This makes it possible to accurately evaluate the customer service attitude of the store staff and the reactions of customers and provide effective feedback.

[0804] "Store staff" refers to employees who serve customers in a store.

[0805] "Role-play videos" are videos filmed by store staff simulating actual customer service situations.

[0806] A "server" is a computer system that receives, processes, stores, and transmits data over a network.

[0807] "Voice recognition technology" refers to technology that converts voice data into text data.

[0808] A "generative AI model" is a model that uses artificial intelligence technology to analyze data and generate results.

[0809] "Text data" refers to text information converted using voice recognition technology.

[0810] "Video data" refers to image information obtained from moving images.

[0811] "Analysis" is the process of examining data in detail and extracting meaning and patterns.

[0812] "Feedback" refers to evaluations and advice provided based on the analysis results.

[0813] "Users" refer to store staff and managers who use this system.

[0814] MODE FOR CARRYING OUT THE INVENTION

[0815] The present invention is a system for improving the customer service skills of store staff, which analyzes role-playing videos and provides feedback. A specific embodiment of this system will be described below.

[0816] System configuration

[0817] Hardware and Software Configuration

[0818] 1. User's device

[0819] Photography devices such as smartphones and digital cameras

[0820] An application for uploading videos to a server

[0821] 2. Server

[0822] A computer system that receives, processes, and analyzes video

[0823] Speech recognition technology (e.g., Google Cloud Speech-to-Text API)

[0824] Generative AI models (e.g., OpenAI's GPT-4)

[0825] Recording and uploading videos

[0826] Users use their smartphones or digital cameras to record role-playing videos of store staff. For example, they can record a scene in which a staff member greets a customer by saying "Welcome." After filming, users select the video file through an application on their device and upload it to the server.

[0827] Video reception and preprocessing

[0828] The server receives the video file uploaded from the device. After receiving it, it checks the format and size of the video file and performs format conversion or compression if necessary. Next, the server extracts the audio portion of the video and converts it to text using speech recognition technology, such as the Google Cloud Speech-to-Text API.

[0829] Video Analysis

[0830] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text and video data in the video. Specifically, it performs the following processes:

[0831] Text data analysis: The server uses natural language processing technology to analyze the text data converted by speech recognition technology and evaluates the content of the store staff's speech. For example, it analyzes the tone and context of the phrase "Welcome."

[0832] Video data analysis: The server uses image recognition technology to analyze the video data and evaluate the facial expressions and gestures of staff and the customer's reactions. For example, it analyzes whether the staff is smiling and the customer's satisfaction level.

[0833] Results generation and feedback

[0834] The server evaluates the customer service skills of store staff based on the analysis results and generates feedback on areas for improvement and merit. For example, it generates specific advice such as, "You don't smile often, so try to smile more." The generated feedback is saved in text format and sent to the device.

[0835] View Feedback

[0836] The terminal displays the feedback received from the server to the user. For example, the evaluation results may be displayed in text or graph format on the application screen. The user can check the displayed feedback and use it to improve their next customer service experience.

[0837] Examples of specific examples and prompts

[0838] Examples:

[0839] Users film role-playing videos on their smartphones and upload them to the server via the application.

[0840] The server receives the video and converts the audio portion into text using the Google Cloud Speech-to-Text API.

[0841] The server uses a generative AI model (OpenAI's GPT-4) to analyze the text and video data, for example evaluating the tone of the staff member's "Welcome" remark and the customer's facial expression.

[0842] Based on the analysis results, the server generates feedback such as "You're not smiling enough, so try to smile more," and sends it to the device.

[0843] The device displays the feedback to the user, who can use it to improve their next customer service experience.

[0844] Example prompt sentence:

[0845] "Please analyze the customer service attitude of the store staff in this video and provide feedback on areas for improvement."

[0846] "Please evaluate the comments made by staff in the video and the reactions of customers, and provide advice to help improve customer service skills."

[0847] In this way, a system is realized in which the server, terminals, and users work together to improve the customer service skills of store staff.

[0848] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0849] Step 1:

[0850] Recording and uploading videos

[0851] The user uses a smartphone or digital camera to record a role-playing video of a store staff member, for example, greeting a customer by saying "Welcome."

[0852] Input: Filmed roleplay video file

[0853] The device provides an application for uploading the captured video to the server. The user selects a video file through the application on the device and presses the upload button.

[0854] Output: Video file uploaded to the server

[0855] Step 2:

[0856] Video reception and preprocessing

[0857] The server receives the video file uploaded from the device, checks the format and size of the video file, and performs format conversion or compression as necessary.

[0858] Input: Uploaded video file

[0859] The server extracts the audio portion of the video and converts the audio into text using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[0860] Output: Text data (text information converted from audio)

[0861] Step 3:

[0862] Video Analysis

[0863] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text and visual data in the video.

[0864] Input: Text data, video data

[0865] The server analyzes the text data using natural language processing technology and evaluates the content of the store staff's comments, such as the tone and context of the phrase "Welcome."

[0866] The server analyzes the video data using image recognition technology to evaluate the facial expressions and gestures of staff and the customer's reactions, for example, whether the staff is smiling or not and the customer's satisfaction level.

[0867] Output: Analysis results (evaluation of speech content, facial expressions and gestures, evaluation of customer reactions)

[0868] Step 4:

[0869] Results generation and feedback

[0870] The server evaluates the customer service skills of store staff based on the analysis results and generates feedback on areas for improvement and strengths. For example, it generates specific advice such as, "You don't smile often, so try to smile more."

[0871] Input: Analysis results

[0872] The server saves the generated feedback in text format and transmits it to the terminal.

[0873] Output: Feedback (textual evaluation and advice)

[0874] Step 5:

[0875] View Feedback

[0876] The terminal displays the feedback received from the server to the user, for example, by displaying the evaluation results in text or graph format on the application screen.

[0877] Input: Feedback

[0878] The user can check the displayed feedback and use it to improve their next customer service experience.

[0879] Output: Feedback displayed to the user (evaluation results and advice)

[0880] In this way, a system is realized in which the server, terminals, and users work together to improve the customer service skills of store staff.

[0881] (Application example 1)

[0882] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0883] Traditional store staff training methods lack the feedback needed to effectively improve their customer service skills. Furthermore, there are limited ways for staff to objectively evaluate their own customer service attitudes and language, making self-improvement difficult. Furthermore, the time and cost required for staff training are also major issues.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0885] In this invention, the server includes a means for analyzing role-playing videos of store staff, a means for store staff to ask questions based on the analysis results, a generative AI means for generating effective suggestion talk in response to the questions, an application means installed on a smartphone, a means for converting video frames into text, and a means for inputting prompts into the generative AI to obtain the analysis results. This allows store staff to objectively evaluate their customer service skills and identify specific areas for improvement. It also improves training efficiency and reduces costs.

[0886] "Store staff" refers to employees who provide products and services to customers in physical stores.

[0887] "Role-play video" is a video that shows store staff simulating customer service.

[0888] The "analysis method" refers to technology that analyzes role-play videos and evaluates the language and attitude of store staff and customer reactions.

[0889] "Generative AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of generating proposed talks.

[0890] The "application means installed on a smartphone" is software that runs on a smartphone and is an application for supporting training of store staff.

[0891] "Means for converting video frames into text" refers to a technology that analyzes each frame of video and converts its content into text format.

[0892] A "prompt" is an instruction to be input to a generative AI, and is text that includes the content of a video frame.

[0893] The "means of obtaining analysis results" refers to the technology of inputting a prompt sentence into a generative AI and obtaining the resulting analysis data.

[0894] The system for implementing this invention analyzes role-playing videos of store staff and provides feedback to improve the staff's customer service skills. The system is implemented using an application installed on a smartphone.

[0895] First, store staff use their smartphones to film a role-playing video. The video is then uploaded to a server via an application. The server analyzes the video frame by frame and converts the content of each frame into text. This text is used as a prompt to input to the generative AI.

[0896] Based on the prompt text, the generative AI analyzes the language and attitude of store staff in the video, as well as customer reactions. The analysis results are provided as specific feedback on the staff's customer service skills, including areas for improvement and specific advice.

[0897] The hardware used is a smartphone, and the software uses Python, OpenCV, and OpenAI API. Specifically, OpenCV is used to divide the video into frames and convert the content of each frame into text. The OpenAI API is used to input prompt text into the generative AI and obtain the analysis results.

[0898] As a concrete example, the following prompt sentence is input into the generative AI.

[0899] Analyze the following video frame:

[0900] Frame 1: Staff members are smiling and greeting customers.

[0901] Frame 2: A staff member explains the product.

[0902] Frame 3: The customer is asking a question.

[0903] Frame 4: Staff member answers questions.

[0904] Frame 5: The customer nods approvingly.

[0905] By inputting this prompt into generative AI, specific feedback on the staff's customer service skills can be obtained. This allows store staff to objectively evaluate their own customer service skills and identify specific areas for improvement. It also leads to more efficient training and reduced costs.

[0906] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0907] Step 1:

[0908] The user shoots a role-playing video using a smartphone. The video is saved in the smartphone. The input is the role-playing video shot by the user, and the output is the video file saved in the smartphone.

[0909] Step 2:

[0910] The device uploads the captured role-play video to a server via the application. The input is the video file stored on the smartphone, and the output is the video file uploaded to the server.

[0911] Step 3:

[0912] The server splits the uploaded video into frames. The input is the video file uploaded to the server, and the output is the image data of each split frame. Specifically, the video is split into frames using OpenCV.

[0913] Step 4:

[0914] The server converts the content of each frame into text. The input is the image data of each divided frame, and the output is text data describing the content of each frame. Specifically, the server uses image recognition technology to analyze the content of the frame and convert it into text format.

[0915] Step 5:

[0916] The server generates a prompt sentence to be input to the generative AI. The input is text data describing the content of each frame, and the output is the generated prompt sentence. Specifically, the text data of each frame is concatenated to create the prompt sentence.

[0917] Step 6:

[0918] The server inputs a prompt to the generative AI and obtains the analysis results. The input is the generated prompt, and the output is the analysis results obtained from the generative AI. Specifically, the server inputs the prompt using OpenAI's API and obtains the analysis data.

[0919] Step 7:

[0920] The server generates feedback for store staff based on the analysis results. The input is the analysis results obtained from the generative AI, and the output is specific feedback content. Specifically, the server analyzes the analysis results and creates feedback that includes areas for improvement and advice regarding the staff's customer service skills.

[0921] Step 8:

[0922] The device receives feedback from the server and displays it to the user. The input is the feedback sent from the server, and the output is the feedback displayed on the smartphone screen. Specifically, the feedback is received through an application and visually presented to the user.

[0923] Example 2

[0924] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0925] Traditional store staff training methods required a lot of time and effort for staff to master effective sales pitches. They also lacked immediate feedback to help staff deliver appropriate sales pitches in real-life situations. This slowed down staff performance improvement and potentially led to lower customer satisfaction.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0927] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, means for the store staff to input questions to the generative AI through an interface, means for sending the input questions to the generative AI model, and means for receiving answers from the generative AI model and displaying them to the store staff. This enables store staff to quickly and effectively learn appropriate suggestion talk and immediately use it in actual situations.

[0928] "Store staff" refers to employees who deal with customers and make product suggestions in stores.

[0929] "Role-play video" refers to footage of store staff simulating customer interactions and product suggestions.

[0930] "Means of analysis" refers to the techniques and methods used to analyze role-play videos and evaluate staff performance and the effectiveness of proposal talks.

[0931] "Means for asking questions" refers to the interface or method that store staff use to input questions about specific situations to the generative AI.

[0932] "Generative AI means" refers to artificial intelligence technology that generates appropriate suggestion talk in response to an input question.

[0933] "Interface" refers to the user interface that store staff use to input questions into the generative AI.

[0934] "Generative AI model" refers to an artificial intelligence model that generates answers based on input prompts.

[0935] "Means of transmission" refers to the communication technology or method for transmitting the input question to the generative AI model.

[0936] "Means for displaying" refers to the technology or method for visually presenting the answers received from the generative AI model to store staff.

[0937] MODE FOR CARRYING OUT THE INVENTION

[0938] This invention is a system that uses generative AI to help store staff learn effective proposal talk. The system works in cooperation with three parties: a server, a terminal, and a user.

[0939] Hardware and software used

[0940] The server is developed using Python and implemented as a web application using the Flask framework. The generative AI model used is OpenAI's GPT-4. The terminals used by store staff are PCs or tablets, and they access the server using a web browser (e.g., Google Chrome).

[0941] Program processing

[0942] The server hosts a web application using the Flask framework and provides an interface for receiving questions from store staff, using HTML and JavaScript to create a user-friendly form for store staff to enter their questions.

[0943] The terminal (store staff's PC or tablet) uses a web browser to access the server and display the interface. The user (store staff) inputs questions to the generative AI through the interface. For example, they might input a question such as, "What is the best suggestion to make in this situation?"

[0944] The entered question is sent to the server via JavaScript. The server passes the received question to the generative AI model. The generative AI model receives the question as a prompt sentence and generates an appropriate answer. The server returns the answer received from the generative AI model to the device, and the device displays the received answer on the interface.

[0945] Specific examples

[0946] For example, consider the case where a store staff member asks, "What is the best suggestion phrase to use when introducing a new product?" This question is sent to the generative AI model as a prompt sentence like the following:

[0947] Example prompt sentence:

[0948] What is the best pitch for introducing a new product?

[0949] The generative AI model generates appropriate suggested speech based on this prompt and sends it back to the server, which then sends the generated suggested speech back to the device and displays it on the interface for the user to review.

[0950] This system enables store staff to quickly and effectively learn appropriate sales pitches and immediately use them in real-life situations.

[0951] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0952] Step 1:

[0953] The user accesses the interface

[0954] The user (store staff) opens a web browser (e.g., Google Chrome) and accesses a specified URL (e.g., http: / / example.com). The server serves the web page using the Flask framework. The input is the user's action of accessing the URL, and the output is the web page on which the interface is displayed.

[0955] Step 2:

[0956] The user enters a question

[0957] The user enters "What is the best suggested talk for this situation?" into the text box on the web page and clicks the submit button. The input is the user's action of entering a question into the text box, and the output is the entered question data.

[0958] Step 3:

[0959] The device sends a question to the server

[0960] The device uses the JavaScript fetch API to send the entered question to the server as an HTTP POST request. The body of the request contains the question entered by the user. The input is the question data entered by the user, and the output is the HTTP POST request sent to the server.

[0961] Step 4:

[0962] The server sends the question to the generative AI model

[0963] The server formats the received question as a prompt sentence and sends it to the generative AI model. The prompt sentence is sent to the generative AI model via an API request. The input is the question data received by the server, and the output is the prompt sentence sent to the generative AI model.

[0964] Step 5:

[0965] Generative AI models generate answers

[0966] The generative AI model generates an answer based on the prompt it receives. For example, in response to the question, "What is the best suggestion for this situation?", it generates an answer such as, "Customer, this product uses the latest technology and is very high-performance." The input is the prompt received by the generative AI model, and the output is the generated answer data.

[0967] Step 6:

[0968] The server receives the generated answer

[0969] The server receives the answer returned from the generative AI model in JSON format. The received JSON data contains the generated answer. The input is the answer data returned from the generative AI model, and the output is the JSON data received by the server.

[0970] Step 7:

[0971] The server sends the answer back to the device

[0972] The server returns the received answer to the terminal as an HTTP response. The body of the response contains the generated answer. The input is the JSON data received by the server, and the output is the HTTP response returned to the terminal.

[0973] Step 8:

[0974] The device displays the answer to the user

[0975] The terminal displays the received answer in a designated area on the interface. The user can check the displayed answer and decide the next action. The input is the HTTP response received by the terminal, and the output is the answer displayed on the interface.

[0976] (Application example 2)

[0977] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0978] Previous systems for improving store staff customer service skills had the problem that analyzing role-play videos and generating suggestion talks took time, making it difficult to provide real-time feedback. Furthermore, there was a lack of a way for store staff to instantly obtain the optimal suggestion talks on the spot, and effective support for improving the quality of customer service was not provided. This could delay the improvement of store staff customer service skills and lead to a decline in customer satisfaction.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0980] In this invention, the server includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, and interactive interface means for store staff to ask questions to the generative AI using a mobile information terminal and obtain optimal suggestion talk in real time. This enables store staff to instantly obtain optimal suggestion talk on the spot, improving the quality of customer service.

[0981] "Store staff" refers to employees who work in stores and handle customer service, product explanations, sales promotion, and other tasks.

[0982] "Role-play video" is a video recording of store staff simulating how to deal with customers.

[0983] "Analysis" refers to the act of evaluating the content of the role-playing videos and measuring the customer service skills of store staff and the effectiveness of their proposal talks.

[0984] "Generative AI" is a system that uses artificial intelligence technology to generate new information and suggestions based on input data.

[0985] A "mobile information terminal" is a portable information processing device such as a smartphone or tablet.

[0986] An "interactive interface" is an operation screen or input means that allows users to interact with the system in a two-way manner.

[0987] "Proposal talk" refers to the way and content of the talk that store staff give to customers to explain products and promote sales.

[0988] "Feedback" refers to evaluations and advice provided by generative AI to store staff to improve their customer service skills.

[0989] A system for implementing this invention includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, and interactive interface means for store staff to ask questions to the generative AI using a mobile information terminal and obtain optimal suggestion talk in real time.

[0990] System configuration

[0991] 1. Hardware:

[0992] Mobile information terminal: A portable information processing device such as a smartphone or tablet.

[0993] Server: A high-performance computer for processing data and running generative AI.

[0994] 2. Software:

[0995] Python: A programming language used to implement the program.

[0996] Tkinter: A GUI library for building user interfaces that run on mobile devices.

[0997] OpenAI API: An API for using generative AI models (e.g., GPT-3).

[0998] Processing flow

[0999] 1. Role-play video analysis:

[1000] The server receives role-playing videos of store staff and uses a video analysis algorithm to evaluate their customer service skills and the effectiveness of their proposal talk.

[1001] The analysis results are stored in a database and used for subsequent processing.

[1002] 2. Generate and submit a question:

[1003] Store staff use mobile information terminals to input questions to the generative AI.

[1004] For example, enter a question such as, "What is the best phrase to use when introducing a new product?"

[1005] 3. Answer generation using generative AI:

[1006] The server sends questions to the generative AI model through the OpenAI API.

[1007] The generative AI model generates the optimal suggested dialogue based on the input question and sends it back to the server.

[1008] 4. Interactive interface display:

[1009] The server transmits the generated proposal talk to a mobile information terminal and displays it to the store staff through an interactive interface.

[1010] Store staff can check the optimal suggestion talk in real time and use it in actual customer service.

[1011] Specific examples

[1012] For example, if a store staff member asks, "What is the best phrase to use when introducing a new product?", the generative AI will provide an answer such as, "This new product uses the latest technology and is extremely high-performance compared to other products. In particular, it features a long battery life and an easy-to-use design."

[1013] Prompt Sentence Examples

[1014] What's the best thing to say when introducing a new product?

[1015] In this way, store staff can instantly receive the most appropriate proposals on-site, improving the quality of customer service.

[1016] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1017] Step 1:

[1018] Role-play video analysis

[1019] The server receives role-playing videos of store staff. The received videos are input into an analysis algorithm to evaluate customer service skills and the effectiveness of the proposed talk. The analysis results are stored in a database and used for subsequent processing. The input is the role-playing video, and the output is the analysis results. Specific operations include frame analysis of the video, voice recognition, and emotion analysis.

[1020] Step 2:

[1021] Generate and submit a question

[1022] The user (store staff) uses a mobile information terminal to input a question to the generative AI. For example, they can input a question such as, "What is the best phrase to use when introducing a new product?" The input is the user's question, and the output is a prompt sent to the generative AI. Specifically, the system accepts text input through the terminal's interface.

[1023] Step 3:

[1024] Answer generation using generative AI

[1025] The server sends a question to the generative AI model via the OpenAI API. The generative AI model generates the optimal suggested dialogue based on the input question and replies to the server. The input is a prompt, and the output is the generated suggested dialogue. Specific operations include sending an API request and receiving a response.

[1026] Step 4:

[1027] Interactive interface display

[1028] The server sends the generated suggested talk to the mobile information terminal and displays it to the user through an interactive interface. The user can check the optimal suggested talk in real time and use it in actual customer service. The input is the generated suggested talk, and the output is the text displayed on the user's terminal. Specific operations include converting the text format and displaying it on the interface.

[1029] Example 3

[1030] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1031] Conventional store staff proposal talks depended on their individual experience and skills, resulting in inconsistency and inconsistency in effectiveness. Furthermore, there was a lack of a feedback loop to generate effective proposal talks, making it difficult to improve staff skills.

[1032] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1033] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, means for presenting the generated suggestion talk to the store staff, means for receiving feedback from the store staff, and means for retraining the generative AI model based on the received feedback. This enables store staff to consistently use effective suggestion talk and continuously improve their skills through feedback.

[1034] "Store staff" refers to employees who deal with customers and engage in sales activities in stores.

[1035] "Role-play video" refers to footage of store staff simulating how to deal with customers.

[1036] "Means of analysis" refers to the techniques and methods used to analyze role-play videos and evaluate the responses and suggestions of store staff.

[1037] "Means of asking questions" refers to the methods and techniques that store staff use to ask appropriate questions to customers.

[1038] "Generative AI methods" refer to technologies and methods for generating effective proposal talks using artificial intelligence.

[1039] "Proposal talk" refers to the content of proposals and explanations given by store staff to customers.

[1040] "Means for presentation" refers to the method or technology for displaying the generated proposal talk to store staff.

[1041] "Means for receiving feedback" refers to methods and techniques for collecting evaluations and opinions from store staff.

[1042] "Means for retraining" refers to techniques and methods for retraining a generative AI model based on received feedback to improve its accuracy.

[1043] MODE FOR CARRYING OUT THE INVENTION

[1044] This invention is a system that allows store staff to generate effective proposal talks and use them in customer service. The system operates mainly with a server, terminals, and users (store staff).

[1045] server

[1046] The server uses a generative AI model to generate appropriate suggestion speech in response to questions from store staff. This generation uses natural language generation technology. Specifically, it utilizes generative AI models such as OpenAI's GPT-4. The server performs the following processes:

[1047] 1. Receive a user question

[1048] The server receives a question from a user (store staff member). For example, the user inputs, "If a customer is looking for a new smartphone, what suggestions should I make?"

[1049] 2. Send a prompt to the generative AI model

[1050] The server sends the received question to the generative AI model as a prompt, which is then converted into a format that the generative AI model can easily understand.

[1051] 3. Receive suggested talks from the generative AI model

[1052] The server receives the proposed speech generated by the generative AI model, such as "This smartphone is equipped with the latest camera functions and is ideal for customers who love photography."

[1053] 4. Send the proposed chat to the device

[1054] The server then sends the generated proposal token to the terminal, which is a device used by the store staff.

[1055] 5. Retraining generative AI models based on feedback

[1056] The server retrains the generative AI model based on the feedback received from the user, allowing the model to generate more effective suggested conversations.

[1057] Terminal

[1058] The terminal receives the suggestion message sent from the server and displays it to the store staff. The terminal is equipped with an interactive interface, allowing the store staff to check the suggestion message and provide feedback as needed.

[1059] 1. Display suggested talks

[1060] The terminal displays the proposed talks sent from the server to the user, and allows the user to check the proposed talks through an interactive interface.

[1061] 2. Receive user feedback

[1062] The device receives feedback provided by the user in response to the proposed talk, such as "This suggestion was effective" or "I would like more specific information."

[1063] User (store staff)

[1064] The user checks the proposed conversations displayed on the device and uses them in actual customer service. In addition, the user provides data to the generative AI to learn from highly successful proposed conversations. The generative AI then uses that data to generate new proposed conversations.

[1065] 1. Check the proposal talk and use it in customer service

[1066] The user can check the suggested conversation displayed on the device and use it in actual customer service. For example, the user might suggest to a customer, "This smartphone is equipped with the latest camera functions, making it ideal for customers who like taking photos."

[1067] 2. Provide feedback

[1068] The user provides feedback on the effectiveness of the proposal talk, for example, by inputting specific feedback such as "This proposal was very well received by the customer."

[1069] Specific examples

[1070] Example prompts to be input to the generative AI model

[1071] "If a customer is looking for a new smartphone, what suggestions should you make?"

[1072] This system allows store staff to always use the latest and most effective suggestion talks, improving the quality of customer service. In addition, the generative AI model continuously learns through feedback, making it possible to improve the accuracy of suggestion talks. The flow of the identification process in Example 3 is explained using Figure 15.

[1073] Step 1:

[1074] The server receives a question from the user (store staff). The user types into the terminal, "If a customer is looking for a new smartphone, what suggestions should I make?" The server receives this question as a prompt. The input is the user's question, and the output is the prompt.

[1075] Step 2:

[1076] The server sends the received prompt to a generative AI model. Specifically, the prompt is input to the generative AI model (e.g., GPT-4) and data processing is performed to generate appropriate suggested speech. The input is the prompt, and the output is the input data for the generative AI model.

[1077] Step 3:

[1078] The server receives the proposed conversation generated by the generative AI model. The generative AI model generates the proposed conversation using natural language generation technology based on the prompt sentence. For example, it might generate something like, "This smartphone is equipped with the latest camera functions and is ideal for customers who love photography." The input is the input data to the generative AI model, and the output is the generated proposed conversation.

[1079] Step 4:

[1080] The server sends the generated suggestion token to the terminal. The terminal is a device used by store staff and has an interface for displaying the suggestion token. The input is the generated suggestion token, and the output is the data sent to the terminal.

[1081] Step 5:

[1082] The terminal receives the proposed tokens sent from the server and displays them to the user. The terminal allows the user to check the proposed tokens through an interactive interface. The input is the data sent from the server, and the output is the data displayed to the user.

[1083] Step 6:

[1084] The user checks the suggestion speech displayed on the device and uses it in actual customer service. For example, a user suggests to a customer, "This smartphone is equipped with the latest camera functions and is perfect for customers who like photography." The input is the suggestion speech displayed on the device, and the output is the suggestion to the customer.

[1085] Step 7:

[1086] The user provides feedback on the effectiveness of the proposal talk. For example, the user inputs specific feedback such as "This proposal was very well received by the customer" into the terminal. The input is information about the effectiveness of the proposal talk, and the output is feedback data.

[1087] Step 8:

[1088] The terminal receives feedback from the user and transmits it to the server. The input is the feedback data from the user, and the output is the data to be transmitted to the server.

[1089] Step 9:

[1090] The server retrains the generative AI model based on the received feedback, allowing the model to generate more effective suggested speeches. The input is the feedback data, and the output is the retrained generative AI model.

[1091] Step 10:

[1092] The server generates new suggested conversations using the retrained generative AI model. This allows the server to provide more accurate suggested conversations for subsequent user questions. The input is the retrained generative AI model, and the output is the newly generated suggested conversation.

[1093] (Application example 3)

[1094] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1095] Conventional store staff proposal conversations depended on their individual experience and skills, and the ability of the staff to make effective proposals was a major factor. Furthermore, there was a lack of real-time support for staff to make appropriate proposals to customers, and a lack of means to receive feedback on the proposal conversations. This led to issues such as insufficient improvement of customer satisfaction and sales promotion.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1097] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective proposal talk in response to the questions, means for presenting the proposal talk generated by the generative AI means to the store staff in real time, and means for recording the proposal talk of the store staff and providing it as feedback later. This enables the store staff to make effective suggestions to customers in real time and also receive feedback to improve the quality of the proposal talk.

[1098] "Store staff" refers to employees who deal with customers and promote sales in physical stores.

[1099] "Role-play video" refers to footage of store staff simulating customer service interactions.

[1100] "Analysis" refers to the process of analyzing role-play videos and other data to extract specific patterns and trends.

[1101] A "question" is a question that a store staff member asks a customer to elicit information.

[1102] "Proposal talk" refers to explanatory and persuasive talk given by store staff to customers in order to recommend products or services.

[1103] "Generative AI" refers to a system that uses artificial intelligence technology to generate natural language and create sentences and conversations for specific purposes.

[1104] "Real-time" refers to a state in which data and information are processed immediately and provided without delay.

[1105] "Recording" refers to the process of saving store staff's suggestion talks and other data for future reference.

[1106] "Feedback" refers to the process of providing improvements and evaluations in response to suggestions made by store staff.

[1107] The system for implementing this invention is configured as follows: First, software for analyzing role-playing videos of store staff is installed on a server. This software uses video analysis technology to extract patterns of staff's proposal talk and customer interactions.

[1108] The server then uses natural language processing technology to generate questions for store staff to ask customers based on the analysis results. The generated questions are then sent to the staff's smartphones in real time.

[1109] Furthermore, the server uses a generative AI model to generate effective proposal speech in response to customer questions. This generative AI model has previously learned the proposal speech of highly successful store staff, and generates new proposal speech based on that knowledge. The generated proposal speech is also displayed in real time on the staff's smartphone.

[1110] The server also records the actual proposal talks made by the store staff and provides the records as feedback later. This feedback includes specific advice for improving the quality of the proposal talks made by the staff.

[1111] In implementing this system, the following hardware and software are used:

[1112] Hardware: Smartphones, servers

[1113] Software: Video analysis software, natural language processing software, generative AI models (e.g., OpenAI GPT)

[1114] As a concrete example, the following prompt sentence is used:

[1115] Learning prompt: Learn from this talk: This product is very popular, especially this time of year.

[1116] Proposal talk generation prompt: Generate a proposal talk for this query: I'd like to know more about this product.

[1117] Feedback prompt: Provide feedback for this talk: This product is very popular, especially this time of year.

[1118] This system allows store staff to make effective suggestions to customers in real time and receive feedback to improve the quality of their suggestions.

[1119] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1120] Step 1:

[1121] The server receives role-playing videos of store staff and analyzes them using video analysis software. The input is the role-playing video, and the output is the analysis results, including the staff's suggestion talk and customer response patterns. Specifically, the system analyzes the video frames and converts the talk content into text using voice recognition technology.

[1122] Step 2:

[1123] The server generates questions for store staff to ask customers based on the analysis results obtained in step 1. The input is the analysis results, and the output is the generated questions. Specifically, it uses natural language processing software to generate appropriate questions and sends them to the staff's smartphones.

[1124] Step 3:

[1125] The server uses a generative AI model to generate effective suggestion speech in response to customer questions. The input is the customer question, and the output is the generated suggestion speech. Specifically, the server inputs a prompt sentence into the generative AI model, and displays the generated speech on the staff member's smartphone in real time.

[1126] Step 4:

[1127] The terminal (smartphone) displays the proposed chat received from the server to the store staff. The input is the proposed chat sent from the server, and the output is the chat displayed on the staff's screen. Specifically, the proposed chat is displayed on the smartphone display so that the staff can check it.

[1128] Step 5:

[1129] The server records the actual proposal conversations made by store staff. The input is the staff's proposal conversations, and the output is the recorded conversation data. Specifically, the server uses voice recognition technology to convert the staff's conversations into text and saves it in a database.

[1130] Step 6:

[1131] The server generates feedback based on the recorded suggestion talk and provides it to the store staff. The input is the recorded talk data, and the output is the generated feedback. Specifically, the server inputs feedback prompts into the generative AI model and sends the generated feedback to the staff's smartphone.

[1132] Step 7:

[1133] The terminal (smartphone) displays the feedback received from the server to the store staff. The input is the feedback sent from the server, and the output is the feedback displayed on the staff's screen. Specifically, the feedback is displayed on the smartphone display so that the staff can check it.

[1134] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1135] "Example 1"

[1136] One embodiment of the present invention provides a system including a means for analyzing role-playing videos of store staff, a means for the store staff to ask questions based on the analysis results, a generative AI means for generating effective suggested talk in response to the questions, and an emotion engine for recognizing user emotions. Specifically, the store staff film role-playing videos simulating customer service scenes and input the videos into the generative AI. The generative AI analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions. Furthermore, the emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI. The generative AI optimizes the suggested talk to the store staff based on this feedback.

[1137] "Example 2"

[1138] As a specific example, consider the case where a store staff member films a role-play video of themselves handling a customer complaint and then inputs that video into a generative AI. The generative AI analyzes the words and attitude of the store staff member in the video, as well as the customer's reaction. Furthermore, the emotion engine analyzes the emotional state of the store staff member and feeds the results back to the generative AI. Based on this feedback, the generative AI optimizes the suggested dialogue to the store staff member. For example, if a store staff member is feeling irritated by a customer complaint, the generative AI will take that emotional state into consideration and generate suggested dialogue to respond more calmly.

[1139] "Example 3"

[1140] One embodiment of the present invention provides a system including a means for analyzing role-playing videos of store staff, a means for the store staff to ask questions based on the analysis results, a generative AI means for generating effective suggested talk in response to the questions, and an emotion engine for recognizing user emotions. Specifically, the store staff film role-playing videos simulating customer service scenes and input the videos into the generative AI. The generative AI analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions. Furthermore, the emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI. The generative AI optimizes the suggested talk to the store staff based on this feedback.

[1141] The processing flow of each embodiment will be described below.

[1142] "Example 1"

[1143] Step 1: Store staff film role-playing videos simulating customer service situations.

[1144] Step 2: Input the captured video into the generative AI.

[1145] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[1146] Step 4: The emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI.

[1147] Step 5: Based on this feedback, the generative AI optimizes the proposal talk to store staff.

[1148] "Example 2"

[1149] Step 1: Store staff film a role-play video of themselves handling a customer complaint. Step 2: The filmed video is input into the generative AI.

[1150] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[1151] Step 4: The emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI.

[1152] Step 5: The generative AI uses this feedback to optimize the suggested conversations to store staff. For example, if a store staff member is feeling irritated by a customer complaint, the generative AI will take that emotional state into account and generate a suggested conversation that will help them respond more calmly.

[1153] Example 1

[1154] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1155] Conventional training methods for improving store staff's customer service skills have made it difficult to effectively analyze staff words, attitudes, and customer reactions and provide feedback. Furthermore, they do not generate suggested conversations that take into account the emotional state of staff, making it difficult to respond effectively in actual customer service situations. This has led to the issue of insufficient improvement of staff customer service skills.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1157] In this invention, the server includes means for shooting role-playing videos of store staff, means for uploading the shot videos to the server, means for analyzing the words, attitudes, and customer reactions in the videos, means for analyzing the emotional state of the store staff, generative AI means for generating suggestive talk based on the results of the emotional analysis, and means for checking the generated suggestive talk and utilizing it in actual customer service, thereby making it possible to effectively improve the customer service skills of store staff.

[1158] "Store staff" refers to employees who serve customers in a store.

[1159] "Role-play videos" are videos filmed by store staff simulating customer service situations.

[1160] A "server" is a computer system for storing, processing, and analyzing data.

[1161] "Upload" is the act of a user sending data from a local device to a server.

[1162] "Generative AI" is a system that uses artificial intelligence technology to analyze data and generate output tailored to specific purposes.

[1163] "Emotional state" refers to changes in the psychological state and emotions of store staff.

[1164] "Proposal talk" refers to the content of proposals and explanations that store staff give to customers during customer service situations.

[1165] "Feedback" is the act of providing improvements and advice based on analysis results and evaluations.

[1166] "Analysis" is the process of examining data in detail to understand its meaning and patterns.

[1167] A "specialized application" is a software program designed for a specific purpose.

[1168] This invention provides a system for improving the customer service skills of store staff, specifically, a technology for analyzing role-play videos and generating suggestive talks. An embodiment of this system will be described in detail below.

[1169] First, the user (store staff) uses a camera such as a smartphone or video camera to shoot a role-play video simulating a customer interaction scene. Once the shooting is complete, the video file is saved.

[1170] Next, the user launches the dedicated application and uploads the saved role-play video to the server. The dedicated application is video uploading software designed to allow users to easily send videos to the server.

[1171] The server receives the uploaded video file and uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the words, attitudes, and customer reactions in the video. The generative AI model converts the audio data in the video into text and performs a detailed analysis of the words, attitudes, and customer reactions.

[1172] Furthermore, the server uses an emotion recognition engine (e.g., Affectiva) to analyze the emotional state of the store staff. The emotion recognition engine analyzes the facial expressions and tone of voice of the store staff in the video and generates numerical data representing their emotional state.

[1173] The server feeds back the results of the emotion analysis to the generative AI to generate the optimal suggestion speech. The generative AI then generates a prompt based on the feedback and creates a suggestion speech for the store staff.

[1174] Users can check the generated suggested conversations through a dedicated application and use them in actual customer service situations, allowing store staff to acquire more effective customer service skills.

[1175] As a concrete example, the following prompt sentence can be input to a generative AI model:

[1176] Example prompt sentence:

[1177] "If a store staff member shows signs of dissatisfaction while serving a customer, please suggest how they should respond."

[1178] This system will enable store staff to improve their ability to respond to customers in actual customer service situations, and is expected to contribute to improving customer satisfaction.

[1179] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1180] Step 1:

[1181] The user uses a smartphone or video camera to shoot a role-play video simulating a customer service scene.

[1182] Input: Mock customer service scene

[1183] Output: Roleplay video file

[1184] Specific operation: The user launches the camera app on their smartphone and starts a role-play simulating a customer service scene. Once the recording is complete, the video file is saved.

[1185] Step 2:

[1186] The user launches the dedicated application and uploads the saved role-play video to the server.

[1187] Input: Roleplay video file

[1188] Output: Video data stored on the server

[1189] Specific operation: The user launches the dedicated application, selects the saved video file, and presses the upload button. The video is then sent to the server.

[1190] Step 3:

[1191] The server uses a generative AI model to analyze the words, attitudes, and customer reactions in the video.

[1192] Input: Video data stored on the server

[1193] Output: Analysis results (words, attitudes, and customer response data)

[1194] How it works: The server receives the uploaded video file and inputs it into a generative AI model. The AI ​​model converts the audio data in the video into text and analyzes the words, attitude, and customer reactions.

[1195] Step 4:

[1196] The server uses an emotion recognition engine to analyze the emotional state of the store staff.

[1197] Input: Analysis results (words, attitudes, customer response data)

[1198] Output: Emotional state data

[1199] Specific operation: The server inputs the facial expressions and tone of voice of the store staff in the video into an emotion recognition engine to analyze their emotional state. The analysis results are generated as numerical data.

[1200] Step 5:

[1201] The server feeds back the results of the emotion analysis to the generative AI to generate the optimal suggested talk.

[1202] Input: Emotional state data

[1203] Output: Proposal Talk

[1204] How it works: The server sends the results of the emotion analysis to the generative AI, which then generates prompts based on the feedback and creates the optimal suggested conversation.

[1205] Step 6:

[1206] Users can check the generated suggestion talk through a dedicated application and use it in actual customer service.

[1207] Input: Suggested Talk

[1208] Output: Improved customer service skills

[1209] Specific operation: The user checks the generated suggestion talk through a dedicated application, reads the suggestion talk, and uses it in actual customer service situations.

[1210] (Application example 1)

[1211] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1212] Conventional training methods for improving store staff's customer service skills have made it difficult to effectively analyze staff language, attitudes, and customer reactions, and provide optimal suggestions. Furthermore, feedback that does not take into account the staff's emotional state is difficult to provide, making it difficult for them to respond effectively in actual customer service situations. This has limited the improvement of staff's customer service skills, and has also limited the improvement of customer satisfaction.

[1213] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1214] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state of the store staff, means for feeding back the results of the emotion engine means to the generative AI means, and means for the generative AI means to generate optimal suggestion talk based on the feedback. This makes it possible to effectively analyze the language and attitude of staff and the reactions of customers, and to provide optimal suggestion talk that takes into account their emotional state.

[1215] "Store staff" refers to employees who provide customer service to customers in physical stores.

[1216] "Role-play videos" are videos filmed by store staff simulating customer service situations.

[1217] "Means of analysis" refers to technology and equipment used to analyze the words and attitudes of store staff and customer reactions in role-play videos.

[1218] "Means for asking questions" refers to the technology or devices that allow store staff to ask appropriate questions to customers based on the analysis results.

[1219] "Generative AI means" refers to technologies and devices that use artificial intelligence technology to generate effective suggestion talk in response to questions.

[1220] The "emotion engine means" refers to a technology or device for analyzing the emotional state of store staff.

[1221] "Feedback means" refers to the technology or device used to transmit the results of the emotion engine means to the generative AI means.

[1222] "Means for generating optimal proposal talk" refers to the technology or device that enables the generative AI means to generate optimal proposal talk for store staff based on feedback.

[1223] A system for implementing this invention analyzes role-play videos and generates optimal proposal talks to improve the customer service skills of store staff. A specific embodiment of this system will be described below.

[1224] System Configuration

[1225] The system consists of the following major components:

[1226] 1. Video recording device: Store staff will record role-play videos using a device with a camera function, such as a smartphone.

[1227] 2. Server: A central processing unit that performs video analysis, sentiment analysis, and generates suggested talks using generative AI.

[1228] 3. Emotion Engine: Software for analyzing the emotional state of store staff.

[1229] 4. Generative AI model: An artificial intelligence model that generates optimal suggested talks based on the results of video analysis and sentiment analysis.

[1230] Video recording and uploading

[1231] Store staff use their smartphones to film their customer service role-playing videos, which are then uploaded to a server via a dedicated application.

[1232] Video Analysis

[1233] The server receives the uploaded video and divides it into frames using a video analysis library such as OpenCV. Each frame is analyzed to determine the language and attitude of store staff and the customer's reaction.

[1234] sentiment analysis

[1235] The server uses an emotion engine to analyze the emotional state of the store staff in each frame. The emotion engine analyzes facial expressions, tone of voice, etc., and quantifies the emotional state of the staff.

[1236] Proposal talk generation

[1237] The server uses a generative AI model to combine the results of video analysis and sentiment analysis to generate optimal suggested talks. The generative AI model learns from past successful suggested talks and provides feedback to staff based on this.

[1238] Providing feedback

[1239] The generated suggestion talk and feedback are provided to store staff via a smartphone application, allowing them to improve their customer service skills.

[1240] Specific examples

[1241] For example, a store staff member uploads a video following the prompt: "Please upload a customer service role-playing video. We will analyze the language, attitude, and customer reactions in the video and provide the optimal suggestion talk." The server analyzes the video and evaluates the staff member's language, attitude, and customer reactions. The emotion engine analyzes the staff member's emotional state and provides feedback to the generative AI. The generative AI then generates the optimal suggestion talk and provides it to the staff member.

[1242] In this way, store staff will be able to respond effectively in actual customer service situations.

[1243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1244] Step 1:

[1245] The user uses a smartphone to film a customer service role-play video.

[1246] Input: Customer Service Roleplay Video

[1247] Output: Recorded video file

[1248] Specific operation: The user starts the camera app on their smartphone and records a role-play simulating a customer service scene. Once the recording is complete, the video file is saved on the smartphone.

[1249] Step 2:

[1250] Users upload videos they have taken to the server using a dedicated application.

[1251] Input: Recorded video file

[1252] Output: Video file uploaded to the server

[1253] Specific operation: The user launches the dedicated application, selects the video file they have taken, and presses the upload button. The application then sends the video file to the server.

[1254] Step 3:

[1255] The server receives the uploaded video and splits it into frames using OpenCV.

[1256] Input: Uploaded video file

[1257] Output: A list of split frame images

[1258] Specific operation: The server receives a video file and splits it into frames using the OpenCV library. Each frame image is saved as a list.

[1259] Step 4:

[1260] The server uses a generative AI model to analyze the language and attitude of store staff and customer reactions in each frame.

[1261] Input: A list of split frame images

[1262] Output: Analysis of language, attitude, and customer response

[1263] How it works: The server uses a generative AI model to analyze each frame image. As a result, the store staff's language and attitude, as well as the customer's reaction, are quantified.

[1264] Step 5:

[1265] The server uses an emotion engine to analyze the emotional state of the store staff in each frame.

[1266] Input: A list of split frame images

[1267] Output: Analysis of store staff's emotional state

[1268] Specific operation: The server uses the emotion engine to analyze the emotional state of the store staff in each frame image. The emotional state is quantified and saved as the analysis result.

[1269] Step 6:

[1270] The server uses a generative AI model to integrate the results of video analysis and sentiment analysis to generate the optimal suggested talk.

[1271] Input: Analysis of language, attitude, customer reactions, and store staff emotional state

[1272] Output: Best suggestion talk

[1273] How it works: The server uses a generative AI model to integrate the results of video analysis and sentiment analysis. Based on the integrated results, the optimal suggested conversation is generated.

[1274] Step 7:

[1275] The server provides the generated proposed talk and feedback to the user through a smartphone application.

[1276] Input: Best Suggestion Talk

[1277] Output: Suggested talk and feedback provided to the user

[1278] Specific operation: The server sends the generated proposed talk and feedback to a dedicated application, and the user receives the proposed talk and feedback through the application.

[1279] The above are the specific processing steps for carrying out the present invention.

[1280] Example 2

[1281] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1282] Conventional store staff training systems lack specific feedback to improve staff response skills. Furthermore, it is difficult to generate suggestive conversations that take into account the emotional state of staff, making it difficult to effectively handle complaints. This can delay the improvement of staff response skills and lead to a decline in customer satisfaction.

[1283] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1284] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state, and means for returning the generated suggestion talk to the store staff. This enables specific feedback to improve the response capabilities of the store staff, and realizes the generation of effective suggestion talk that takes the emotional state into consideration.

[1285] "Store staff" refers to employees who deal with customers and perform sales duties in a store.

[1286] A "role-play video" is a video in which store staff act out specific situations and are filmed.

[1287] "Means of analysis" refers to the technology and equipment used to analyze the content of role-play videos and evaluate the behavior and words of store staff and the reactions of customers.

[1288] The "means of asking questions" refers to an interface or system that allows store staff to input questions into the generative AI and obtain information based on those questions.

[1289] "Generative AI means" refers to technologies or devices that use artificial intelligence technology to generate optimal proposal talks based on input data.

[1290] "Emotion engine means" refers to technology or equipment that analyzes the emotional state of people in a video and feeds the results back to the generative AI.

[1291] "Suggestion talk" refers to the optimal response or comments that store staff should make to customers in a specific situation.

[1292] "Means of providing feedback" refers to the technology and devices used to communicate the proposed talk and analysis results generated by generative AI to store staff.

[1293] This invention is a system for improving the customer service skills of store staff, specifically, analyzing role-play videos and generating optimal proposal talks. A specific embodiment of this system will be described below.

[1294] First, the user (store staff) uses a device such as a smartphone or tablet to film a role-play video of handling a complaint. This video records the store staff acting out a specific situation.

[1295] Next, the user accesses the server through a web browser or a dedicated application. The interface includes a video upload function and a question input field. The user uploads the role-play video they have filmed to the server through the interface.

[1296] The server receives the uploaded video and inputs it into a generative AI model. This generative AI model uses OpenAI's GPT-4 or Google Cloud Vision API, for example, to analyze the words and attitudes of store staff in the video, as well as customer reactions. Specifically, it uses natural language processing (NLP) technology to analyze the content of conversations, and image recognition technology to analyze facial expressions and gestures.

[1297] The server then uses an emotion engine to analyze the emotional state of the store staff in the video. The emotion engine can use, for example, Microsoft Azure's Emotion API. The analysis results of the emotion engine are fed back to the generative AI model.

[1298] The generative AI model uses this feedback to generate optimal suggested dialogue. For example, if a store staff member appears frustrated in the video, the generative AI model takes that emotion into account and generates specific suggested dialogue for responding calmly.

[1299] Finally, the server returns the generated proposed token to the user, who can then check it through the interface and use it in their work.

[1300] As a concrete example, consider the case where a user inputs the following prompt sentence into a generative AI model:

[1301] Example prompt sentence:

[1302] "We have filmed a role-play video of us handling a customer complaint. Please analyze this video and tell us the best way to present our proposal."

[1303] In this way, users can receive suggestions from the generative AI and use them in their work. This system enables specific feedback to improve the responsiveness of store staff, and generates effective suggestion conversations that take into account their emotional state.

[1304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1305] Step 1:

[1306] The user shoots a role-play video.

[1307] Input: A situation in which a user takes a photo using a device such as a smartphone or tablet.

[1308] Specific operation: The user acts out a situation in which they are handling a complaint and takes a picture of the situation using the device's camera.

[1309] Output: The captured role-play video file.

[1310] Step 2:

[1311] A user uploads a video to a server.

[1312] Input: Filmed roleplay video file.

[1313] Specific operation: Users access the interface through a web browser or dedicated application, select a video file, and upload it.

[1314] Output: Roleplay video files stored on the server.

[1315] Step 3:

[1316] The server inputs the video into a generative AI model.

[1317] Input: Roleplay video files stored on the server.

[1318] Specific operation: The server converts the video file into the appropriate format and inputs it into the generative AI model.

[1319] Output: Video data fed into the generative AI model.

[1320] Step 4:

[1321] A generative AI model analyzes the video.

[1322] Input: Video data fed into the generative AI model.

[1323] How it works: The generative AI model uses natural language processing (NLP) techniques to analyze conversational content and image recognition techniques to analyze facial expressions and gestures, such as OpenAI's GPT-4 and Google Cloud Vision API.

[1324] Output: Analysis results of the words and attitudes of store staff in the video and customer reactions.

[1325] Step 5:

[1326] The emotion engine analyzes the emotional state.

[1327] Input: Analysis results of the words and attitudes of store staff in the video and customer reactions.

[1328] How it works: The emotion engine uses, for example, Microsoft Azure's Emotion API to identify emotions from the facial expressions and tone of voice of store staff.

[1329] Output: Analysis of store staff's emotional state.

[1330] Step 6:

[1331] A generative AI model generates suggested talks.

[1332] Input: Analysis results on the emotional state of store staff.

[1333] Specific behavior: The generative AI model generates optimal suggestion speech based on feedback from the emotion engine. For example, if a store staff member is feeling irritated, it takes that emotion into account and generates specific suggestion speech to respond calmly.

[1334] Output: The generated proposal talk.

[1335] Step 7:

[1336] The server returns the generated suggested token to the user.

[1337] Input: The generated proposal token.

[1338] Specific operation: The server receives the proposed talk generated from the generative AI model and returns it to the user through the interface.

[1339] Output: A suggested talk that the user can see.

[1340] In this way, users can receive suggestions from generative AI and use them in their actual work.

[1341] (Application example 2)

[1342] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1343] In the past, it was difficult for store staff to generate optimal suggestions that took into account the emotional state of the staff and the customer's reaction when handling complaints. Furthermore, there was a lack of means for staff to reflect on their own responses and receive concrete feedback to improve. This could delay the improvement of staff's response skills and lead to a decline in customer satisfaction.

[1344] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1345] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state, and means for feeding back the results of the emotion engine means to the generative AI means. This makes it possible to generate optimal suggestion talk that takes into account the emotional state of the store staff and the reactions of customers, which is expected to improve the staff's response skills and customer satisfaction.

[1346] "Store staff" refers to employees who deal with customers and perform sales duties in physical stores.

[1347] "Role-play video" refers to footage of store staff simulating how to deal with customers.

[1348] "Means of analysis" refers to the technology and equipment used to analyze the content of role-play videos and evaluate the behavior and words of store staff and the reactions of customers.

[1349] The "means of asking questions" refers to the interface or method that store staff use to input specific questions to the generative AI.

[1350] "Generative AI means" refers to artificial intelligence that generates optimal proposal talks based on input data.

[1351] "Emotion engine means" refers to technology or equipment that analyzes the emotional state of store staff in videos and feeds the results back to the generative AI.

[1352] "Feedback means" refers to the methods and technologies for transmitting the analysis results of the emotion engine means to the generative AI means.

[1353] "Suggestion talk" refers to the optimal words and phrases that store staff can use when dealing with customers.

[1354] A system for implementing this invention includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing emotional states, and means for feeding back the results of the emotion engine means to the generative AI means.

[1355] System configuration

[1356] The system consists of the following main components:

[1357] 1. Smartphone: A device used by store staff to record role-play videos of customer complaint handling and to run applications.

[1358] 2. Server: A central processing unit that analyzes videos, analyzes emotional states, and generates suggested talks using generative AI.

[1359] 3. Generative AI model: An artificial intelligence model for generating proposed conversations. Specifically, we use a natural language generation model such as GPT-2.

[1360] 4. Emotion Engine: A model for analyzing the emotional state of store staff in videos. Specifically, we use a sentiment analysis model using TensorFlow.

[1361] Video analysis and emotional state analysis

[1362] The server receives role-play videos uploaded from smartphones. The videos are analyzed frame by frame, and the store staff's behavior and customer reactions are evaluated. The emotion engine preprocesses each frame in the video and predicts the store staff's emotional state. This makes it clear what emotional state the staff is in when responding.

[1363] Proposal talk generation

[1364] The generative AI model receives feedback from the emotion engine and generates optimal suggestions that take into account the emotional state of the store staff and the customer's reaction. For example, if a store staff member is in an irritated emotional state and the customer is reacting angrily, the generative AI model will generate suggestions such as "understand the customer's feelings and offer an apology."

[1365] Specific examples

[1366] As a specific use case, consider the case where a store staff member asks, "If a customer is angry, what is a suggested conversation to respond calmly?" A role-play video taken with a smartphone is uploaded to the server, and the emotion engine analyzes the staff member's emotional state. The generative AI model generates a suggested conversation based on the following prompt:

[1367] If a store staff member is in an irritated emotional state and a customer is reacting angrily, what is the best suggestion talk?

[1368] In this way, store staff can reflect on their own responses and identify specific areas for improvement, which is expected to improve staff response skills and customer satisfaction.

[1369] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1370] Step 1:

[1371] The user (store staff) uses a smartphone to shoot a role-play video of handling a complaint. The video is saved on the smartphone.

[1372] Step 2:

[1373] The user starts the application on their smartphone and uploads the role-playing video they have taken to the server. The input is the role-playing video, and the output is the video data sent to the server.

[1374] Step 3:

[1375] The server analyzes the received role-play video. Specifically, it divides the video into frames and preprocesses the image data of each frame. The input is the video data, and the output is the preprocessed frame data.

[1376] Step 4:

[1377] The server inputs the preprocessed frame data into the emotion engine to analyze the emotional state of the store staff. The emotion engine predicts the emotional state from the image data of each frame and generates a result. The input is the preprocessed frame data, and the output is the predicted emotional state.

[1378] Step 5:

[1379] The server feeds back the emotional state prediction results obtained from the emotion engine to the generative AI model. The input is the emotional state prediction results, and the output is feedback data to the generative AI model.

[1380] Step 6:

[1381] A user inputs a question to the generative AI model through a smartphone application. For example, they input a question like, "If a customer is angry, what is a suggested conversation to respond calmly?" The input is the user's question, and the output is prompt data for the generative AI model.

[1382] Step 7:

[1383] The generative AI model generates optimal suggested speech based on the predicted emotional state and the user's question. The input is the predicted emotional state and prompt data, and the output is the suggested speech.

[1384] Step 8:

[1385] The server sends the generated proposed token to the smartphone application. The input is the proposed token, and the output is the data to be sent to the smartphone.

[1386] Step 9:

[1387] The user can then review the generated suggested speech through a smartphone application, allowing them to reflect on their own actions and identify specific areas for improvement. The input is the suggested speech, and the output is the user's understanding and feedback.

[1388] Example 3

[1389] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1390] Previous systems for improving store staff customer service skills had the problem of being difficult to generate suggestive conversations and provide feedback that fully considered the skills and emotional state of each staff member. They also lacked a means to effectively convey the know-how of highly successful staff members to other staff members. This resulted in inconsistencies in the quality of customer service, making it difficult to improve overall customer satisfaction.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1392] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggested talk in response to the questions, means for presenting the generated suggested talk to the store staff, emotion engine means for recognizing the emotional state of the store staff, means for feeding back the results of the emotion engine means to the generative AI means, and means for providing feedback to the store staff based on the suggested talk generated by the generative AI means. This makes it possible to generate and provide feedback on optimal suggested talk that takes into account the individual skills and emotional states of store staff, thereby improving the quality of customer service and customer satisfaction.

[1393] "Store staff" refers to employees who handle customer service and product explanations in stores.

[1394] "Role-play videos" are videos filmed by store staff simulating customer service scenes.

[1395] The "analysis method" refers to the technology used to analyze the content of role-play videos and evaluate the behavior of store staff and customer reactions.

[1396] The "means for asking questions" refers to a technique that enables store staff to generate appropriate questions based on the analysis results and ask them to customers.

[1397] "Generative AI methods" are technologies that use artificial intelligence to generate effective proposals in response to questions from store staff.

[1398] "Proposal talk" refers to the product explanations and proposals made by store staff to customers.

[1399] The "emotion engine means" is a technology for recognizing the emotional state of store staff and analyzing that information.

[1400] The "means of providing feedback" is a technology that provides specific advice and guidance to store staff based on the proposed talk generated by the generative AI means.

[1401] "Highly performing store staff" are store staff who demonstrate excellent results in evaluation criteria such as customer satisfaction and sales performance.

[1402] The "learning means" is a technology that allows the generative AI means to learn the proposal talk of store staff who are achieving high results.

[1403] MODE FOR CARRYING OUT THE INVENTION

[1404] This invention is a system for improving the customer service skills of store staff, using a generative AI model and an emotion engine to generate effective suggestion talk and provide feedback.

[1405] System configuration

[1406] This system consists of three main components: a server, a terminal, and a user.

[1407] server

[1408] The server is equipped with a generative AI model and an emotion engine. The generative AI model uses natural language generation technology to generate appropriate suggestion speech in response to store staff questions. Specifically, the server uses machine learning technology to learn the suggestion speech of highly successful store staff and uses the results to generate new suggestion speech. In addition, the emotion engine recognizes the emotional state of store staff and feeds the results back to the generative AI model.

[1409] Terminal

[1410] The terminal is a device used by store staff, and displays the proposed talk generated by the generative AI through an interactive interface. In addition, the terminal also records role-playing videos of the store staff and sends the videos to a server.

[1411] User

[1412] The user (store staff) films a role-playing video simulating a customer service situation and sends it to the server via their device. The server analyzes the words, attitudes, and customer reactions in the video, and uses an emotion engine to recognize the store staff's emotional state. The results are fed back to the generative AI, which generates an optimized proposal conversation.

[1413] Hardware and software used

[1414] Hardware: Servers, smartphones, tablets

[1415] Software: Generative AI models (e.g., GPT-4), emotion engines, video analysis software (e.g., OpenCV), speech recognition software (e.g., Google Speech-to-Text)

[1416] Specific examples

[1417] As a concrete example, consider a scene in which a store staff member is introducing a new product. The user films the introduction of the new product as a role-play video and sends it to the server via their device. The server analyzes the video, and the generative AI generates an appropriate suggestion dialogue. For example, the generated suggestion dialogue might be, "This new product is especially popular with the younger generation. Would you like to try it?"

[1418] Prompt Sentence Examples

[1419] "Please film a role-play video introducing a new product and send it to the server. The generative AI will generate an appropriate proposal speech."

[1420] This system enables store staff to serve customers more effectively, and is expected to improve customer satisfaction. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1421] Step 1:

[1422] The user shoots a role-play video.

[1423] Specific operation: The user uses a smartphone or tablet to record role-play videos that simulate customer service scenes, such as introducing a new product or interacting with a customer.

[1424] Input: Customer service scene demonstration

[1425] Output: Roleplay video file

[1426] Step 2:

[1427] The video captured by the device is sent to the server.

[1428] Specific operation: Open the application on your device and tap the "Upload video" button. Select the video you have taken and send it to the server.

[1429] Input: Roleplay video file

[1430] Output: Video data uploaded to the server

[1431] Step 3:

[1432] The server analyzes the video.

[1433] Specific operation: The server analyzes the video using video analysis software (e.g., OpenCV), and converts the audio data into text using speech recognition software (e.g., Google Speech-to-Text).

[1434] Input: Uploaded video data

[1435] Output: Text data and analysis results

[1436] Step 4:

[1437] The server generates proposed talks using generative AI.

[1438] Specific operation: The server generates a suggested conversation using a generative AI model (e.g., GPT-4). The generated text is saved as a suggested conversation.

[1439] Input: Text data and analysis results

[1440] Output: Text data of the proposed talk

[1441] Step 5:

[1442] The server sends the generated proposed token to the terminal.

[1443] Specific operation: The server sends the generated proposed token to the device application, and the sent data is notified to the user through the device's notification function.

[1444] Input: Text data of the proposed talk

[1445] Output: Proposal token sent to the device

[1446] Step 6:

[1447] The terminal displays the suggested talk to the user.

[1448] Specific operation: Open the application on your device and check the suggested talk. The suggested talk will be displayed on the main screen of the application.

[1449] Input: Suggested token sent to the device

[1450] Output: Suggested speech displayed on the screen

[1451] Step 7:

[1452] The user provides customer service based on the proposed talk.

[1453] Specific operation: The user provides actual customer service based on the displayed suggestion talk. For example, the user introduces a product to a customer using the generated suggestion talk.

[1454] Input: Suggested speech displayed on the screen

[1455] Output: Actual customer service scene

[1456] Step 8:

[1457] The server learns the customer service results and updates the generative AI.

[1458] Specific operation: The server collects the results of customer service interactions and trains the generative AI. Specifically, it collects the success rate of interactions and customer reaction data and updates the generative AI model.

[1459] Input: Customer service result data

[1460] Output: An updated generative AI model

[1461] (Application example 3)

[1462] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1463] Conventional training methods for improving store staff's customer service skills make it difficult to properly evaluate the emotional state of individual staff members and the quality of their specific proposal talks, and provide feedback. Furthermore, there is a lack of specific means for generating effective proposal talks, which means it takes time to improve staff skills. This has led to the issue of delays in improving the quality of customer service across the entire store.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1465] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state of the store staff, and means for providing feedback based on the generated suggestion talk and the emotional state, thereby making it possible to efficiently improve the customer service skills of store staff.

[1466] "Store staff" refers to employees who serve customers and make sales in physical stores.

[1467] "Role-play videos" refer to videos filmed by store staff simulating customer service situations.

[1468] "Means of analysis" refers to the technology and equipment used to analyze role-play videos and evaluate the words and attitudes of store staff, customer reactions, etc.

[1469] "Generative AI methods" refer to systems and programs that use artificial intelligence technology to generate effective suggestion talk in response to questions from store staff.

[1470] "Emotion engine means" refers to technology or devices that analyze the emotional state of store staff and provide feedback on the results to generative AI.

[1471] "Means for providing feedback" refers to a system or program for providing specific advice or guidance to store staff based on the generated suggestion talk and emotional state.

[1472] A system for carrying out this invention has the following configuration: A server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, emotion engine means for analyzing the emotional state of the store staff, and means for providing feedback based on the generated suggestion talk and emotional state.

[1473] Hardware and software used

[1474] Hardware: Smartphone (with camera)

[1475] Software: OpenCV (video processing), Transformers (GPT-3 model), EmotionRecognizer (emotion recognition)

[1476] Processing flow

[1477] 1. Loading a roleplay video:

[1478] Role-playing videos taken with a smartphone are uploaded to the server, which uses OpenCV to split the videos into frames and prepare them for analysis.

[1479] 2. Video Analysis:

[1480] The server analyzes the words and attitudes of store staff and customer reactions in the video, specifically using facial and voice recognition to extract the staff's facial expressions and what they say.

[1481] 3. Proposal talk generation:

[1482] The server uses the GPT-3 model to generate effective suggestions based on a customer service scenario. For example, it uses the following prompt sentence as input:

[1483] "Hello, customer. What product are you looking for today?"

[1484] 4. Sentiment Analysis:

[1485] The server uses EmotionRecognizer to analyze the emotional state of the store staff in the video, thereby understanding the emotional state of the staff when serving customers.

[1486] 5. Providing Feedback:

[1487] The server then provides specific advice and guidance to store staff based on the generated suggested speech and their emotional state, such as advice on how to improve the suggested speech or how to serve customers according to their emotional state.

[1488] Specific examples

[1489] For example, when a store staff member is serving a customer based on the prompt "Hello, customer. What kind of product are you looking for today?", the server generates the following suggestion dialogue:

[1490] "Hello, customer. What products would you like to purchase today? If you're not sure what you're looking for, let me introduce you to some new products, especially for this season."

[1491] Furthermore, if the emotion engine analyzes a staff member's emotional state as "tense," the server will provide feedback such as "Try to speak in a relaxed manner."

[1492] In this way, store staff can develop more effective customer service skills.

[1493] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1494] Step 1:

[1495] The server receives role-playing videos taken with a smartphone and splits the videos into frames using OpenCV. The input is a role-playing video file, and the output is a list of frames. Specifically, the server reads the video file, extracts each frame sequentially, and stores them in a list.

[1496] Step 2:

[1497] The server performs facial and voice recognition for each frame to extract the facial expressions and speech of the store staff. The input is a list of frames, and the output is a list of facial and voice recognition results. Specifically, it uses a facial recognition algorithm to detect faces in each frame, and a voice recognition engine to convert the speech into text.

[1498] Step 3:

[1499] The server uses the GPT-3 model to generate effective suggestion speech based on the customer service scenario. The input is the prompt sentence and the speech recognition result, and the output is the generated suggestion speech. Specifically, the prompt sentence is input into the GPT-3 model, and the generated text is obtained as the suggestion speech.

[1500] Step 4:

[1501] The server uses EmotionRecognizer to analyze the emotional states of store staff in the video. The input is a list of face recognition results, and the output is a list of emotional states. Specifically, the face images in each frame are input to EmotionRecognizer, and the emotional state is estimated.

[1502] Step 5:

[1503] The server provides specific advice and guidance to store staff based on the generated suggested speech and emotional state. The input is a list of suggested speech and emotional state, and the output is a feedback message. Specifically, the server evaluates the content of the suggested speech and the emotional state, and generates a feedback message containing suggestions for improvement and advice.

[1504] Step 6:

[1505] The server sends the generated feedback message to the store staff's smartphone. The input is the feedback message, and the output is a notification to the staff's smartphone. Specifically, the server uses a message sending API to send the feedback message to the smartphone.

[1506] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1507] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1508] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1509] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1510] [Third embodiment]

[1511] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1512] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1513] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1514] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1515] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1516] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1517] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1518] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1519] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1520] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1521] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1522] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1523] "Example 1"

[1524] This embodiment of the present invention uses video analysis technology as a means of analyzing role-playing videos of store staff. Specifically, store staff film role-playing videos simulating actual customer service scenes, and input the videos into a generative AI. The generative AI then analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions.

[1525] "Example 2"

[1526] Next, an interactive interface is provided as a means for store staff to ask questions to the generative AI. Specifically, store staff can ask the generative AI questions such as, "What is the best suggestion to make in this situation?"

[1527] "Example 3"

[1528] The generative AI then uses natural language generation technology to generate effective suggestion speeches. Specifically, the generative AI generates appropriate suggestion speeches in response to questions from store staff and presents them to them.

[1529] "Example 4"

[1530] Furthermore, machine learning technology is used to train the generative AI to learn the proposal speech of highly successful store staff. Specifically, the generative AI learns the proposal speech of highly successful store staff, and the results are used to generate new proposal speech.

[1531] "Example 5"

[1532] Finally, the interactive interface is used again to provide feedback to store staff based on the proposed speech learned by the generative AI. Specifically, the system provides specific advice and guidance to store staff based on the proposed speech generated by the generative AI.

[1533] The processing flow of each embodiment will be described below.

[1534] "Example 1"

[1535] Step 1: Store staff film role-playing videos simulating actual customer service situations.

[1536] Step 2: Input the filmed role-play video into the generative AI.

[1537] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[1538] "Example 2"

[1539] Step 1: Store staff ask questions to the generative AI through an interactive interface, such as, "What is the best suggestion to make in this situation?"

[1540] "Example 3"

[1541] Step 1: The generative AI generates an appropriate suggestion speech in response to the store staff's question. Step 2: The generated suggestion speech is presented to the store staff.

[1542] "Example 4"

[1543] Step 1: Have the generative AI learn the proposal talk of highly successful store staff.

[1544] Step 2: The generative AI uses the learning results to generate new proposed talks.

[1545] "Example 5"

[1546] Step 1: Based on the proposed conversation generated by the generative AI, specific advice and guidance is provided to store staff.

[1547] Example 1

[1548] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1549] Conventional training methods for improving store staff's customer service skills make it difficult to accurately evaluate the staff's actual customer service attitude and customer reactions, making it impossible to provide effective feedback.In addition, there was a lack of concrete measures to improve the quality of staff's suggestion talk, which limited the improvement in customer service skills.

[1550] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1551] In this invention, the server includes means for shooting role-playing videos of store staff, means for uploading the shot videos to the server, means for receiving the uploaded videos and converting the speech into text using speech recognition technology, means for analyzing the text data and video data in the videos using a generative AI model, means for evaluating the customer service skills of the store staff based on the analysis results and generating feedback, and means for displaying the generated feedback to the user. This makes it possible to accurately evaluate the customer service attitude of the store staff and the reactions of customers and provide effective feedback.

[1552] "Store staff" refers to employees who serve customers in a store.

[1553] "Role-play videos" are videos filmed by store staff simulating actual customer service situations.

[1554] A "server" is a computer system that receives, processes, stores, and transmits data over a network.

[1555] "Voice recognition technology" refers to technology that converts voice data into text data.

[1556] A "generative AI model" is a model that uses artificial intelligence technology to analyze data and generate results.

[1557] "Text data" refers to text information converted using voice recognition technology.

[1558] "Video data" refers to image information obtained from moving images.

[1559] "Analysis" is the process of examining data in detail and extracting meaning and patterns.

[1560] "Feedback" refers to evaluations and advice provided based on the analysis results.

[1561] "Users" refer to store staff and managers who use this system.

[1562] MODE FOR CARRYING OUT THE INVENTION

[1563] The present invention is a system for improving the customer service skills of store staff, which analyzes role-playing videos and provides feedback. A specific embodiment of this system will be described below.

[1564] System configuration

[1565] Hardware and Software Configuration

[1566] 1. User's device

[1567] Photography devices such as smartphones and digital cameras

[1568] An application for uploading videos to a server

[1569] 2. Server

[1570] A computer system that receives, processes, and analyzes video

[1571] Speech recognition technology (e.g., Google Cloud Speech-to-Text API)

[1572] Generative AI models (e.g., OpenAI's GPT-4)

[1573] Recording and uploading videos

[1574] Users use their smartphones or digital cameras to record role-playing videos of store staff. For example, they can record a scene in which a staff member greets a customer by saying "Welcome." After filming, users select the video file through an application on their device and upload it to the server.

[1575] Video reception and preprocessing

[1576] The server receives the video file uploaded from the device. After receiving it, it checks the format and size of the video file and performs format conversion or compression if necessary. Next, the server extracts the audio portion of the video and converts it to text using speech recognition technology, such as the Google Cloud Speech-to-Text API.

[1577] Video Analysis

[1578] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text and video data in the video. Specifically, it performs the following processes:

[1579] Text data analysis: The server uses natural language processing technology to analyze the text data converted by speech recognition technology and evaluates the content of the store staff's speech. For example, it analyzes the tone and context of the phrase "Welcome."

[1580] Video data analysis: The server uses image recognition technology to analyze the video data and evaluate the facial expressions and gestures of staff and the customer's reactions. For example, it analyzes whether the staff is smiling and the customer's satisfaction level.

[1581] Results generation and feedback

[1582] The server evaluates the customer service skills of store staff based on the analysis results and generates feedback on areas for improvement and merit. For example, it generates specific advice such as, "You don't smile often, so try to smile more." The generated feedback is saved in text format and sent to the device.

[1583] View Feedback

[1584] The terminal displays the feedback received from the server to the user. For example, the evaluation results may be displayed in text or graph format on the application screen. The user can check the displayed feedback and use it to improve their next customer service experience.

[1585] Examples of specific examples and prompts

[1586] Examples:

[1587] Users film role-playing videos on their smartphones and upload them to the server via the application.

[1588] The server receives the video and converts the audio portion into text using the Google Cloud Speech-to-Text API.

[1589] The server uses a generative AI model (OpenAI's GPT-4) to analyze the text and video data, for example evaluating the tone of the staff member's "Welcome" remark and the customer's facial expression.

[1590] Based on the analysis results, the server generates feedback such as "You're not smiling enough, so try to smile more," and sends it to the device.

[1591] The device displays the feedback to the user, who can use it to improve their next customer service experience.

[1592] Example prompt sentence:

[1593] "Please analyze the customer service attitude of the store staff in this video and provide feedback on areas for improvement."

[1594] "Please evaluate the comments made by staff in the video and the reactions of customers, and provide advice to help improve customer service skills."

[1595] In this way, a system is realized in which the server, terminals, and users work together to improve the customer service skills of store staff.

[1596] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1597] Step 1:

[1598] Recording and uploading videos

[1599] The user uses a smartphone or digital camera to record a role-playing video of a store staff member, for example, greeting a customer by saying "Welcome."

[1600] Input: Filmed roleplay video file

[1601] The device provides an application for uploading the captured video to the server. The user selects a video file through the application on the device and presses the upload button.

[1602] Output: Video file uploaded to the server

[1603] Step 2:

[1604] Video reception and preprocessing

[1605] The server receives the video file uploaded from the device, checks the format and size of the video file, and performs format conversion or compression as necessary.

[1606] Input: Uploaded video file

[1607] The server extracts the audio portion of the video and converts the audio into text using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[1608] Output: Text data (text information converted from audio)

[1609] Step 3:

[1610] Video Analysis

[1611] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text and visual data in the video.

[1612] Input: Text data, video data

[1613] The server analyzes the text data using natural language processing technology and evaluates the content of the store staff's comments, such as the tone and context of the phrase "Welcome."

[1614] The server analyzes the video data using image recognition technology to evaluate the facial expressions and gestures of staff and the customer's reactions, for example, whether the staff is smiling or not and the customer's satisfaction level.

[1615] Output: Analysis results (evaluation of speech content, facial expressions and gestures, evaluation of customer reactions)

[1616] Step 4:

[1617] Results generation and feedback

[1618] The server evaluates the customer service skills of store staff based on the analysis results and generates feedback on areas for improvement and strengths. For example, it generates specific advice such as, "You don't smile often, so try to smile more."

[1619] Input: Analysis results

[1620] The server saves the generated feedback in text format and transmits it to the terminal.

[1621] Output: Feedback (textual evaluation and advice)

[1622] Step 5:

[1623] View Feedback

[1624] The terminal displays the feedback received from the server to the user, for example, by displaying the evaluation results in text or graph format on the application screen.

[1625] Input: Feedback

[1626] The user can check the displayed feedback and use it to improve their next customer service experience.

[1627] Output: Feedback displayed to the user (evaluation results and advice)

[1628] In this way, a system is realized in which the server, terminals, and users work together to improve the customer service skills of store staff.

[1629] (Application example 1)

[1630] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1631] Traditional store staff training methods lack the feedback needed to effectively improve their customer service skills. Furthermore, there are limited ways for staff to objectively evaluate their own customer service attitudes and language, making self-improvement difficult. Furthermore, the time and cost required for staff training are also major issues.

[1632] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1633] In this invention, the server includes a means for analyzing role-playing videos of store staff, a means for store staff to ask questions based on the analysis results, a generative AI means for generating effective suggestion talk in response to the questions, an application means installed on a smartphone, a means for converting video frames into text, and a means for inputting prompts into the generative AI to obtain the analysis results. This allows store staff to objectively evaluate their customer service skills and identify specific areas for improvement. It also improves training efficiency and reduces costs.

[1634] "Store staff" refers to employees who provide products and services to customers in physical stores.

[1635] "Role-play video" is a video that shows store staff simulating customer service.

[1636] The "analysis method" refers to technology that analyzes role-play videos and evaluates the language and attitude of store staff and customer reactions.

[1637] "Generative AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of generating proposed talks.

[1638] The "application means installed on a smartphone" is software that runs on a smartphone and is an application for supporting training of store staff.

[1639] "Means for converting video frames into text" refers to a technology that analyzes each frame of video and converts its content into text format.

[1640] A "prompt" is an instruction to be input to a generative AI, and is text that includes the content of a video frame.

[1641] The "means of obtaining analysis results" refers to the technology of inputting a prompt sentence into a generative AI and obtaining the resulting analysis data.

[1642] The system for implementing this invention analyzes role-playing videos of store staff and provides feedback to improve the staff's customer service skills. The system is implemented using an application installed on a smartphone.

[1643] First, store staff use their smartphones to film a role-playing video. The video is then uploaded to a server via an application. The server analyzes the video frame by frame and converts the content of each frame into text. This text is used as a prompt to input to the generative AI.

[1644] Based on the prompt text, the generative AI analyzes the language and attitude of store staff in the video, as well as customer reactions. The analysis results are provided as specific feedback on the staff's customer service skills, including areas for improvement and specific advice.

[1645] The hardware used is a smartphone, and the software uses Python, OpenCV, and OpenAI API. Specifically, OpenCV is used to divide the video into frames and convert the content of each frame into text. The OpenAI API is used to input prompt text into the generative AI and obtain the analysis results.

[1646] As a concrete example, the following prompt sentence is input into the generative AI.

[1647] Analyze the following video frame:

[1648] Frame 1: Staff members are smiling and greeting customers.

[1649] Frame 2: A staff member explains the product.

[1650] Frame 3: The customer is asking a question.

[1651] Frame 4: Staff member answers questions.

[1652] Frame 5: The customer nods approvingly.

[1653] By inputting this prompt into generative AI, specific feedback on the staff's customer service skills can be obtained. This allows store staff to objectively evaluate their own customer service skills and identify specific areas for improvement. It also leads to more efficient training and reduced costs.

[1654] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1655] Step 1:

[1656] The user shoots a role-playing video using a smartphone. The video is saved in the smartphone. The input is the role-playing video shot by the user, and the output is the video file saved in the smartphone.

[1657] Step 2:

[1658] The device uploads the captured role-play video to a server via the application. The input is the video file stored on the smartphone, and the output is the video file uploaded to the server.

[1659] Step 3:

[1660] The server splits the uploaded video into frames. The input is the video file uploaded to the server, and the output is the image data of each split frame. Specifically, the video is split into frames using OpenCV.

[1661] Step 4:

[1662] The server converts the content of each frame into text. The input is the image data of each divided frame, and the output is text data describing the content of each frame. Specifically, the server uses image recognition technology to analyze the content of the frame and convert it into text format.

[1663] Step 5:

[1664] The server generates a prompt sentence to be input to the generative AI. The input is text data describing the content of each frame, and the output is the generated prompt sentence. Specifically, the text data of each frame is concatenated to create the prompt sentence.

[1665] Step 6:

[1666] The server inputs a prompt to the generative AI and obtains the analysis results. The input is the generated prompt, and the output is the analysis results obtained from the generative AI. Specifically, the server inputs the prompt using OpenAI's API and obtains the analysis data.

[1667] Step 7:

[1668] The server generates feedback for store staff based on the analysis results. The input is the analysis results obtained from the generative AI, and the output is specific feedback content. Specifically, the server analyzes the analysis results and creates feedback that includes areas for improvement and advice regarding the staff's customer service skills.

[1669] Step 8:

[1670] The device receives feedback from the server and displays it to the user. The input is the feedback sent from the server, and the output is the feedback displayed on the smartphone screen. Specifically, the feedback is received through an application and visually presented to the user.

[1671] Example 2

[1672] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1673] Traditional store staff training methods required a lot of time and effort for staff to master effective sales pitches. They also lacked immediate feedback to help staff deliver appropriate sales pitches in real-life situations. This slowed down staff performance improvement and potentially led to lower customer satisfaction.

[1674] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1675] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, means for the store staff to input questions to the generative AI through an interface, means for sending the input questions to the generative AI model, and means for receiving answers from the generative AI model and displaying them to the store staff. This enables store staff to quickly and effectively learn appropriate suggestion talk and immediately use it in actual situations.

[1676] "Store staff" refers to employees who deal with customers and make product suggestions in stores.

[1677] "Role-play video" refers to footage of store staff simulating customer interactions and product suggestions.

[1678] "Means of analysis" refers to the techniques and methods used to analyze role-play videos and evaluate staff performance and the effectiveness of proposal talks.

[1679] "Means for asking questions" refers to the interface or method that store staff use to input questions about specific situations to the generative AI.

[1680] "Generative AI means" refers to artificial intelligence technology that generates appropriate suggestion talk in response to an input question.

[1681] "Interface" refers to the user interface that store staff use to input questions into the generative AI.

[1682] "Generative AI model" refers to an artificial intelligence model that generates answers based on input prompts.

[1683] "Means of transmission" refers to the communication technology or method for transmitting the input question to the generative AI model.

[1684] "Means for displaying" refers to the technology or method for visually presenting the answers received from the generative AI model to store staff.

[1685] MODE FOR CARRYING OUT THE INVENTION

[1686] This invention is a system that uses generative AI to help store staff learn effective proposal talk. The system works in cooperation with three parties: a server, a terminal, and a user.

[1687] Hardware and software used

[1688] The server is developed using Python and implemented as a web application using the Flask framework. The generative AI model used is OpenAI's GPT-4. The terminals used by store staff are PCs or tablets, and they access the server using a web browser (e.g., Google Chrome).

[1689] Program processing

[1690] The server hosts a web application using the Flask framework and provides an interface for receiving questions from store staff, using HTML and JavaScript to create a user-friendly form for store staff to enter their questions.

[1691] The terminal (store staff's PC or tablet) uses a web browser to access the server and display the interface. The user (store staff) inputs questions to the generative AI through the interface. For example, they might input a question such as, "What is the best suggestion to make in this situation?"

[1692] The entered question is sent to the server via JavaScript. The server passes the received question to the generative AI model. The generative AI model receives the question as a prompt sentence and generates an appropriate answer. The server returns the answer received from the generative AI model to the device, and the device displays the received answer on the interface.

[1693] Specific examples

[1694] For example, consider the case where a store staff member asks, "What is the best suggestion phrase to use when introducing a new product?" This question is sent to the generative AI model as a prompt sentence like the following:

[1695] Example prompt sentence:

[1696] What is the best pitch for introducing a new product?

[1697] The generative AI model generates appropriate suggested speech based on this prompt and sends it back to the server, which then sends the generated suggested speech back to the device and displays it on the interface for the user to review.

[1698] This system enables store staff to quickly and effectively learn appropriate sales pitches and immediately use them in real-life situations.

[1699] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1700] Step 1:

[1701] The user accesses the interface

[1702] The user (store staff) opens a web browser (e.g., Google Chrome) and accesses a specified URL (e.g., http: / / example.com). The server serves the web page using the Flask framework. The input is the user's action of accessing the URL, and the output is the web page on which the interface is displayed.

[1703] Step 2:

[1704] The user enters a question

[1705] The user enters "What is the best suggested talk for this situation?" into the text box on the web page and clicks the submit button. The input is the user's action of entering a question into the text box, and the output is the entered question data.

[1706] Step 3:

[1707] The device sends a question to the server

[1708] The device uses the JavaScript fetch API to send the entered question to the server as an HTTP POST request. The body of the request contains the question entered by the user. The input is the question data entered by the user, and the output is the HTTP POST request sent to the server.

[1709] Step 4:

[1710] The server sends the question to the generative AI model

[1711] The server formats the received question as a prompt sentence and sends it to the generative AI model. The prompt sentence is sent to the generative AI model via an API request. The input is the question data received by the server, and the output is the prompt sentence sent to the generative AI model.

[1712] Step 5:

[1713] Generative AI models generate answers

[1714] The generative AI model generates an answer based on the prompt it receives. For example, in response to the question, "What is the best suggestion for this situation?", it generates an answer such as, "Customer, this product uses the latest technology and is very high-performance." The input is the prompt received by the generative AI model, and the output is the generated answer data.

[1715] Step 6:

[1716] The server receives the generated answer

[1717] The server receives the answer returned from the generative AI model in JSON format. The received JSON data contains the generated answer. The input is the answer data returned from the generative AI model, and the output is the JSON data received by the server.

[1718] Step 7:

[1719] The server sends the answer back to the device

[1720] The server returns the received answer to the terminal as an HTTP response. The body of the response contains the generated answer. The input is the JSON data received by the server, and the output is the HTTP response returned to the terminal.

[1721] Step 8:

[1722] The device displays the answer to the user

[1723] The terminal displays the received answer in a designated area on the interface. The user can check the displayed answer and decide the next action. The input is the HTTP response received by the terminal, and the output is the answer displayed on the interface.

[1724] (Application example 2)

[1725] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1726] Previous systems for improving store staff customer service skills had the problem that analyzing role-play videos and generating suggestion talks took time, making it difficult to provide real-time feedback. Furthermore, there was a lack of a way for store staff to instantly obtain the optimal suggestion talks on the spot, and effective support for improving the quality of customer service was not provided. This could delay the improvement of store staff customer service skills and lead to a decline in customer satisfaction.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1728] In this invention, the server includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, and interactive interface means for store staff to ask questions to the generative AI using a mobile information terminal and obtain optimal suggestion talk in real time. This enables store staff to instantly obtain optimal suggestion talk on the spot, improving the quality of customer service.

[1729] "Store staff" refers to employees who work in stores and handle customer service, product explanations, sales promotion, and other tasks.

[1730] "Role-play video" is a video recording of store staff simulating how to deal with customers.

[1731] "Analysis" refers to the act of evaluating the content of the role-playing videos and measuring the customer service skills of store staff and the effectiveness of their proposal talks.

[1732] "Generative AI" is a system that uses artificial intelligence technology to generate new information and suggestions based on input data.

[1733] A "mobile information terminal" is a portable information processing device such as a smartphone or tablet.

[1734] An "interactive interface" is an operation screen or input means that allows users to interact with the system in a two-way manner.

[1735] "Proposal talk" refers to the way and content of the talk that store staff give to customers to explain products and promote sales.

[1736] "Feedback" refers to evaluations and advice provided by generative AI to store staff to improve their customer service skills.

[1737] A system for implementing this invention includes means for analyzing role-playing videos of store staff, means for store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, and interactive interface means for store staff to ask questions to the generative AI using a mobile information terminal and obtain optimal suggestion talk in real time.

[1738] System configuration

[1739] 1. Hardware:

[1740] Mobile information terminal: A portable information processing device such as a smartphone or tablet.

[1741] Server: A high-performance computer for processing data and running generative AI.

[1742] 2. Software:

[1743] Python: A programming language used to implement the program.

[1744] Tkinter: A GUI library for building user interfaces that run on mobile devices.

[1745] OpenAI API: An API for using generative AI models (e.g., GPT-3).

[1746] Processing flow

[1747] 1. Role-play video analysis:

[1748] The server receives role-playing videos of store staff and uses a video analysis algorithm to evaluate their customer service skills and the effectiveness of their proposal talk.

[1749] The analysis results are stored in a database and used for subsequent processing.

[1750] 2. Generate and submit a question:

[1751] Store staff use mobile information terminals to input questions to the generative AI.

[1752] For example, enter a question such as, "What is the best phrase to use when introducing a new product?"

[1753] 3. Answer generation using generative AI:

[1754] The server sends questions to the generative AI model through the OpenAI API.

[1755] The generative AI model generates the optimal suggested dialogue based on the input question and sends it back to the server.

[1756] 4. Interactive interface display:

[1757] The server transmits the generated proposal talk to a mobile information terminal and displays it to the store staff through an interactive interface.

[1758] Store staff can check the optimal suggestion talk in real time and use it in actual customer service.

[1759] Specific examples

[1760] For example, if a store staff member asks, "What is the best phrase to use when introducing a new product?", the generative AI will provide an answer such as, "This new product uses the latest technology and is extremely high-performance compared to other products. In particular, it features a long battery life and an easy-to-use design."

[1761] Prompt Sentence Examples

[1762] What's the best thing to say when introducing a new product?

[1763] In this way, store staff can instantly receive the most appropriate proposals on-site, improving the quality of customer service.

[1764] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1765] Step 1:

[1766] Role-play video analysis

[1767] The server receives role-playing videos of store staff. The received videos are input into an analysis algorithm to evaluate customer service skills and the effectiveness of the proposed talk. The analysis results are stored in a database and used for subsequent processing. The input is the role-playing video, and the output is the analysis results. Specific operations include frame analysis of the video, voice recognition, and emotion analysis.

[1768] Step 2:

[1769] Generate and submit a question

[1770] The user (store staff) uses a mobile information terminal to input a question to the generative AI. For example, they can input a question such as, "What is the best phrase to use when introducing a new product?" The input is the user's question, and the output is a prompt sent to the generative AI. Specifically, the system accepts text input through the terminal's interface.

[1771] Step 3:

[1772] Answer generation using generative AI

[1773] The server sends a question to the generative AI model via the OpenAI API. The generative AI model generates the optimal suggested dialogue based on the input question and replies to the server. The input is a prompt, and the output is the generated suggested dialogue. Specific operations include sending an API request and receiving a response.

[1774] Step 4:

[1775] Interactive interface display

[1776] The server sends the generated suggested talk to the mobile information terminal and displays it to the user through an interactive interface. The user can check the optimal suggested talk in real time and use it in actual customer service. The input is the generated suggested talk, and the output is the text displayed on the user's terminal. Specific operations include converting the text format and displaying it on the interface.

[1777] Example 3

[1778] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1779] Conventional store staff proposal talks depended on their individual experience and skills, resulting in inconsistency and inconsistency in effectiveness. Furthermore, there was a lack of a feedback loop to generate effective proposal talks, making it difficult to improve staff skills.

[1780] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1781] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective suggestion talk in response to the questions, means for presenting the generated suggestion talk to the store staff, means for receiving feedback from the store staff, and means for retraining the generative AI model based on the received feedback. This enables store staff to consistently use effective suggestion talk and continuously improve their skills through feedback.

[1782] "Store staff" refers to employees who deal with customers and engage in sales activities in stores.

[1783] "Role-play video" refers to footage of store staff simulating how to deal with customers.

[1784] "Means of analysis" refers to the techniques and methods used to analyze role-play videos and evaluate the responses and suggestions of store staff.

[1785] "Means of asking questions" refers to the methods and techniques that store staff use to ask appropriate questions to customers.

[1786] "Generative AI methods" refer to technologies and methods for generating effective proposal talks using artificial intelligence.

[1787] "Proposal talk" refers to the content of proposals and explanations given by store staff to customers.

[1788] "Means for presentation" refers to the method or technology for displaying the generated proposal talk to store staff.

[1789] "Means for receiving feedback" refers to methods and techniques for collecting evaluations and opinions from store staff.

[1790] "Means for retraining" refers to techniques and methods for retraining a generative AI model based on received feedback to improve its accuracy.

[1791] MODE FOR CARRYING OUT THE INVENTION

[1792] This invention is a system that allows store staff to generate effective proposal talks and use them in customer service. The system operates mainly with a server, terminals, and users (store staff).

[1793] server

[1794] The server uses a generative AI model to generate appropriate suggestion speech in response to questions from store staff. This generation uses natural language generation technology. Specifically, it utilizes generative AI models such as OpenAI's GPT-4. The server performs the following processes:

[1795] 1. Receive a user question

[1796] The server receives a question from a user (store staff member). For example, the user inputs, "If a customer is looking for a new smartphone, what suggestions should I make?"

[1797] 2. Send a prompt to the generative AI model

[1798] The server sends the received question to the generative AI model as a prompt, which is then converted into a format that the generative AI model can easily understand.

[1799] 3. Receive suggested talks from the generative AI model

[1800] The server receives the proposed speech generated by the generative AI model, such as "This smartphone is equipped with the latest camera functions and is ideal for customers who love photography."

[1801] 4. Send the proposed chat to the device

[1802] The server then sends the generated proposal token to the terminal, which is a device used by the store staff.

[1803] 5. Retraining generative AI models based on feedback

[1804] The server retrains the generative AI model based on the feedback received from the user, allowing the model to generate more effective suggested conversations.

[1805] Terminal

[1806] The terminal receives the suggestion message sent from the server and displays it to the store staff. The terminal is equipped with an interactive interface, allowing the store staff to check the suggestion message and provide feedback as needed.

[1807] 1. Display suggested talks

[1808] The terminal displays the proposed talks sent from the server to the user, and allows the user to check the proposed talks through an interactive interface.

[1809] 2. Receive user feedback

[1810] The device receives feedback provided by the user in response to the proposed talk, such as "This suggestion was effective" or "I would like more specific information."

[1811] User (store staff)

[1812] The user checks the proposed conversations displayed on the device and uses them in actual customer service. In addition, the user provides data to the generative AI to learn from highly successful proposed conversations. The generative AI then uses that data to generate new proposed conversations.

[1813] 1. Check the proposal talk and use it in customer service

[1814] The user can check the suggested conversation displayed on the device and use it in actual customer service. For example, the user might suggest to a customer, "This smartphone is equipped with the latest camera functions, making it ideal for customers who like taking photos."

[1815] 2. Provide feedback

[1816] The user provides feedback on the effectiveness of the proposal talk, for example, by inputting specific feedback such as "This proposal was very well received by the customer."

[1817] Specific examples

[1818] Example prompts to be input to the generative AI model

[1819] "If a customer is looking for a new smartphone, what suggestions should you make?"

[1820] This system allows store staff to always use the latest and most effective suggestion talks, improving the quality of customer service. In addition, the generative AI model continuously learns through feedback, making it possible to improve the accuracy of suggestion talks. The flow of the identification process in Example 3 is explained using Figure 15.

[1821] Step 1:

[1822] The server receives a question from the user (store staff). The user types into the terminal, "If a customer is looking for a new smartphone, what suggestions should I make?" The server receives this question as a prompt. The input is the user's question, and the output is the prompt.

[1823] Step 2:

[1824] The server sends the received prompt to a generative AI model. Specifically, the prompt is input to the generative AI model (e.g., GPT-4) and data processing is performed to generate appropriate suggested speech. The input is the prompt, and the output is the input data for the generative AI model.

[1825] Step 3:

[1826] The server receives the proposed conversation generated by the generative AI model. The generative AI model generates the proposed conversation using natural language generation technology based on the prompt sentence. For example, it might generate something like, "This smartphone is equipped with the latest camera functions and is ideal for customers who love photography." The input is the input data to the generative AI model, and the output is the generated proposed conversation.

[1827] Step 4:

[1828] The server sends the generated suggestion token to the terminal. The terminal is a device used by store staff and has an interface for displaying the suggestion token. The input is the generated suggestion token, and the output is the data sent to the terminal.

[1829] Step 5:

[1830] The terminal receives the proposed tokens sent from the server and displays them to the user. The terminal allows the user to check the proposed tokens through an interactive interface. The input is the data sent from the server, and the output is the data displayed to the user.

[1831] Step 6:

[1832] The user checks the suggestion speech displayed on the device and uses it in actual customer service. For example, a user suggests to a customer, "This smartphone is equipped with the latest camera functions and is perfect for customers who like photography." The input is the suggestion speech displayed on the device, and the output is the suggestion to the customer.

[1833] Step 7:

[1834] The user provides feedback on the effectiveness of the proposal talk. For example, the user inputs specific feedback such as "This proposal was very well received by the customer" into the terminal. The input is information about the effectiveness of the proposal talk, and the output is feedback data.

[1835] Step 8:

[1836] The terminal receives feedback from the user and transmits it to the server. The input is the feedback data from the user, and the output is the data to be transmitted to the server.

[1837] Step 9:

[1838] The server retrains the generative AI model based on the received feedback, allowing the model to generate more effective suggested speeches. The input is the feedback data, and the output is the retrained generative AI model.

[1839] Step 10:

[1840] The server generates new suggested conversations using the retrained generative AI model. This allows the server to provide more accurate suggested conversations for subsequent user questions. The input is the retrained generative AI model, and the output is the newly generated suggested conversation.

[1841] (Application example 3)

[1842] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1843] Conventional store staff proposal conversations depended on their individual experience and skills, and the ability of the staff to make effective proposals was a major factor. Furthermore, there was a lack of real-time support for staff to make appropriate proposals to customers, and a lack of means to receive feedback on the proposal conversations. This led to issues such as insufficient improvement of customer satisfaction and sales promotion.

[1844] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1845] In this invention, the server includes means for analyzing role-playing videos of store staff, means for the store staff to ask questions based on the analysis results, generative AI means for generating effective proposal talk in response to the questions, means for presenting the proposal talk generated by the generative AI means to the store staff in real time, and means for recording the proposal talk of the store staff and providing it as feedback later. This enables the store staff to make effective suggestions to customers in real time and also receive feedback to improve the quality of the proposal talk.

[1846] "Store staff" refers to employees who deal with customers and promote sales in physical stores.

[1847] "Role-play video" refers to footage of store staff simulating customer service interactions.

[1848] "Analysis" refers to the process of analyzing role-play videos and other data to extract specific patterns and trends.

[1849] A "question" is a question that a store staff member asks a customer to elicit information.

[1850] "Proposal talk" refers to explanatory and persuasive talk given by store staff to customers in order to recommend products or services.

[1851] "Generative AI" refers to a system that uses artificial intelligence technology to generate natural language and create sentences and conversations for specific purposes.

[1852] "Real-time" refers to a state in which data and information are processed immediately and provided without delay.

[1853] "Recording" refers to the process of saving store staff's suggestion talks and other data for future reference.

[1854] "Feedback" refers to the process of providing improvements and evaluations in response to suggestions made by store staff.

[1855] The system for implementing this invention is configured as follows: First, software for analyzing role-playing videos of store staff is installed on a server. This software uses video analysis technology to extract patterns of staff's proposal talk and customer interactions.

[1856] The server then uses natural language processing technology to generate questions for store staff to ask customers based on the analysis results. The generated questions are then sent to the staff's smartphones in real time.

[1857] Furthermore, the server uses a generative AI model to generate effective proposal speech in response to customer questions. This generative AI model has previously learned the proposal speech of highly successful store staff, and generates new proposal speech based on that knowledge. The generated proposal speech is also displayed in real time on the staff's smartphone.

[1858] The server also records the actual proposal talks made by the store staff and provides the records as feedback later. This feedback includes specific advice for improving the quality of the proposal talks made by the staff.

[1859] In implementing this system, the following hardware and software are used:

[1860] Hardware: Smartphones, servers

[1861] Software: Video analysis software, natural language processing software, generative AI models (e.g., OpenAI GPT)

[1862] As a concrete example, the following prompt sentence is used:

[1863] Learning prompt: Learn from this talk: This product is very popular, especially this time of year.

[1864] Proposal talk generation prompt: Generate a proposal talk for this query: I'd like to know more about this product.

[1865] Feedback prompt: Provide feedback for this talk: This product is very popular, especially this time of year.

[1866] This system allows store staff to make effective suggestions to customers in real time and receive feedback to improve the quality of their suggestions.

[1867] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1868] Step 1:

[1869] The server receives role-playing videos of store staff and analyzes them using video analysis software. The input is the role-playing video, and the output is the analysis results, including the staff's suggestion talk and customer response patterns. Specifically, the system analyzes the video frames and converts the talk content into text using voice recognition technology.

[1870] Step 2:

[1871] The server generates questions for store staff to ask customers based on the analysis results obtained in step 1. The input is the analysis results, and the output is the generated questions. Specifically, it uses natural language processing software to generate appropriate questions and sends them to the staff's smartphones.

[1872] Step 3:

[1873] The server uses a generative AI model to generate effective suggestion speech in response to customer questions. The input is the customer question, and the output is the generated suggestion speech. Specifically, the server inputs a prompt sentence into the generative AI model, and displays the generated speech on the staff member's smartphone in real time.

[1874] Step 4:

[1875] The terminal (smartphone) displays the proposed chat received from the server to the store staff. The input is the proposed chat sent from the server, and the output is the chat displayed on the staff's screen. Specifically, the proposed chat is displayed on the smartphone display so that the staff can check it.

[1876] Step 5:

[1877] The server records the actual proposal conversations made by store staff. The input is the staff's proposal conversations, and the output is the recorded conversation data. Specifically, the server uses voice recognition technology to convert the staff's conversations into text and saves it in a database.

[1878] Step 6:

[1879] The server generates feedback based on the recorded suggestion talk and provides it to the store staff. The input is the recorded talk data, and the output is the generated feedback. Specifically, the server inputs feedback prompts into the generative AI model and sends the generated feedback to the staff's smartphone.

[1880] Step 7:

[1881] The terminal (smartphone) displays the feedback received from the server to the store staff. The input is the feedback sent from the server, and the output is the feedback displayed on the staff's screen. Specifically, the feedback is displayed on the smartphone display so that the staff can check it.

[1882] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1883] "Example 1"

[1884] One embodiment of the present invention provides a system including a means for analyzing role-playing videos of store staff, a means for the store staff to ask questions based on the analysis results, a generative AI means for generating effective suggested talk in response to the questions, and an emotion engine for recognizing user emotions. Specifically, the store staff film role-playing videos simulating customer service scenes and input the videos into the generative AI. The generative AI analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions. Furthermore, the emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI. The generative AI optimizes the suggested talk to the store staff based on this feedback.

[1885] "Example 2"

[1886] As a specific example, consider the case where a store staff member films a role-play video of themselves handling a customer complaint and then inputs that video into a generative AI. The generative AI analyzes the words and attitude of the store staff member in the video, as well as the customer's reaction. Furthermore, the emotion engine analyzes the emotional state of the store staff member and feeds the results back to the generative AI. Based on this feedback, the generative AI optimizes the suggested dialogue to the store staff member. For example, if a store staff member is feeling irritated by a customer complaint, the generative AI will take that emotional state into consideration and generate suggested dialogue to respond more calmly.

[1887] "Example 3"

[1888] One embodiment of the present invention provides a system including a means for analyzing role-playing videos of store staff, a means for the store staff to ask questions based on the analysis results, a generative AI means for generating effective suggested talk in response to the questions, and an emotion engine for recognizing user emotions. Specifically, the store staff film role-playing videos simulating customer service scenes and input the videos into the generative AI. The generative AI analyzes the words and attitudes of the store staff in the videos, as well as the customer's reactions. Furthermore, the emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI. The generative AI optimizes the suggested talk to the store staff based on this feedback.

[1889] The processing flow of each embodiment will be described below.

[1890] "Example 1"

[1891] Step 1: Store staff film role-playing videos simulating customer service situations.

[1892] Step 2: Input the captured video into the generative AI.

[1893] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[1894] Step 4: The emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI.

[1895] Step 5: Based on this feedback, the generative AI optimizes the proposal talk to store staff.

[1896] "Example 2"

[1897] Step 1: Store staff film a role-play video of themselves handling a customer complaint. Step 2: The filmed video is input into the generative AI.

[1898] Step 3: The generative AI analyzes the words and attitudes of store staff in the video, as well as customer reactions.

[1899] Step 4: The emotion engine analyzes the emotional state of the store staff and feeds the results back to the generative AI.

[1900] Step 5: The generative AI uses this feedback to optimize the suggested conversations to store staff. For example, if a store staff member is feeling irritated by a customer complaint, the generative AI will take that emotional state into account and generate a suggested conversation that will help them respond more calmly.

[1901] Example 1

[1902] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1903] Conventional training methods for improving store staff's customer service skills have made it difficult to effectively analyze staff words, attitudes, and customer reactions and provide feedback. Furthermore, they do not generate suggested conversations that take into account the emotional state of staff, making it difficult to respond effectively in actual customer service situations. This has led to the issue of insufficient improvement of staff customer service skills.

[1904] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1905] In this invention, the server includes means for shooting role-playing videos of store staff, means for uploading the shot videos to the server, means for analyzing the words, attitudes, and customer reactions in the videos, means for analyzing the emotional state of the store staff, generative AI means for generating suggestive talk based on the results of the emotional analysis, and means for checking the generated suggestive talk and utilizing it in actual customer service, thereby making it possible to effectively improve the customer service skills of store staff.

[1906] "Store staff" refers to employees who serve customers in a store.

[1907] "Role-play videos" are videos filmed by store staff simulating customer service situations.

[1908] A "server" is a computer system for storing, processing, and analyzing data.

[1909] "Upload" is the act of a user sending data from a local device to a server.

[1910] "Generative AI" is a system that uses artificial intelligence technology to analyze data and generate output tailored to specific purposes.

[1911] "Emotional state" refers to changes in the psychological state and emotions of store staff.

[1912] "Proposal talk" refers to the content of proposals and explanations that store staff give to customers during customer service situations.

[1913] "Feedback" is the act of providing improvements and advice based on analysis results and evaluations.

[1914] "Analysis" is the process of examining data in detail to understand its meaning and patterns.

[1915] A "specialized application" is a software program designed for a specific purpose.

[1916] This invention provides a system for improving the customer service skills of store staff, specifically, a technology for analyzing role-play videos and generating suggestive talks. An embodiment of this system will be described in detail below.

[1917] First, the user (store staff) uses a camera such as a smartphone or video camera to...

Claims

1. A means to shoot role-playing videos of store staff, A means to upload the recorded video to the server, A means of analyzing the words and attitudes in the video and the customer's reactions, A means for analyzing the emotional state of said store staff; A generation system AI means for generating a proposal talk based on the result of the emotion analysis of the store staff; A way to check the generated proposal talk and use it in actual customer service, A system including:

2. The system according to claim 1 , wherein the generative AI means further includes means for learning suggested talks of store staff who have achieved high results.

3. The system of claim 1 , wherein the generative AI means further comprises means for providing feedback to store staff based on the learned suggestion talk.

Images