System

Abstract

To provide a system.SOLUTION: A system includes: means for acquiring audio data of role playing of a new employee; means for converting the acquired audio data into character information; means for generating a prompt sentence instructing evaluations with respect to evaluation items on the basis of the character information; and means for creating a feedback using the prompt sentence and a generative AI model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] When training new crew members, there is an issue that the feedback given varies depending on the sales department in charge, making it difficult to standardize training. [Means for solving the problem]

[0005] To solve this problem, we will record the role-playing of new recruits, transcribe it, and input it into a generative AI. Furthermore, we will provide a system that systematizes the feedback content as a professional role in sales training, and simultaneously trains sales representatives and crew members. [Brief description of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. The processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit),

[0010] Examples of such processors include a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0011] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0012] In the following embodiments, the storage with the code is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g.,

[0013] Examples include hard disks, or magnetic tapes.

[0014] In the following embodiments, the communication I / F (Interface) with the code is a communication processor and

[0015] The communication I / F is an interface including a power supply, a power source, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0016] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0017] [First embodiment]

[0018] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0019] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0020] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0021] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0022] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0023] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0024] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0025] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0026] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0027] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0028] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0029] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0030] "Example 1"

[0031] In one embodiment of the present invention, a device is provided for recording the role-playing of new crew members. This device obtains voice data of the role-playing using, for example, a microphone or a recording function of a smartphone.

[0032] "Example 2"

[0033] Next, the acquired voice data is converted into text information. This conversion is performed, for example, using voice recognition technology. Specifically, the voice recognition API of Google (registered trademark) or IBM's Watson (registered trademark) is used.

[0034] A voice recognition service may be used.

[0035] "Example 3"

[0036] The converted text information is then input into a generative AI, which is an AI that can mimic human conversation and generate natural conversation based on the input text information.

[0037] "Example 4"

[0038] Finally, the feedback content of the generated conversation is systematized as a professional role in sales training. This systematization can be done, for example, using text analysis and natural language processing technology.

[0039] The process flow of each embodiment will be described below.

[0040] "Example 1"

[0041] Step 1: The new crew member begins role-playing. This role-playing is based on a sales scenario, in which the new crew member simulates interactions with a customer.

[0042] Step 2: Once the role-playing session has started, record it using a microphone or smartphone recording function. This recording is done throughout the role-playing session, capturing in detail the comments and reactions of the new crew member.

[0043] "Example 2"

[0044] Step 1: Once the recording is complete, the audio data is converted into text information. This conversion is done using voice recognition technology. Specifically, it uses voice recognition services such as Google's voice recognition API or IBM's Watson.

[0045] Step 2: The speech recognition service converts the speech data into text data and outputs the result, which is used in the following steps.

[0046] "Example 3"

[0047] Step 1: The text data output from the speech recognition service is input into a generative AI, which is an AI that can imitate human conversation.

[0048] Step 2: The generative AI generates natural conversations based on the input text data. This generated conversation is used as feedback to new crew members.

[0049] "Example 4"

[0050] Step 1: Systematize the feedback content generated by the generative AI as a professional role in sales training. This systematization is done using text analysis and natural language processing technology.

[0051] Step 2: Structured feedback is provided to new crew members to help them improve their performance. This feedback also serves as a key input for developing both salespeople and crew members.

[0052] Example 1

[0053] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0054] In conventional role-playing for new crew members, evaluation and feedback of the recorded content was often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of consistency and objectivity in the evaluation, making effective training difficult. Furthermore, the recording data was not stored or analyzed sufficiently, and the data was not fully utilized.

[0055] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0056] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for simultaneously training sales representatives and crew, a means for storing the recorded data in a cloud server, a means for analyzing the stored voice data, and a means for providing the evaluation results from the generative AI model to the user. This enables automatic analysis of the recorded content and provision of feedback, improving the consistency and objectivity of evaluation and realizing effective training. In addition, the recording data is efficiently stored and analyzed, promoting the utilization of data.

[0057] "Means for recording the role-playing content of new crew members" refers to a device or function for acquiring audio of the role-playing performed by new crew members.

[0058] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[0059] "Means of inputting the transcribed content into the generative AI model" refers to the process or function by which the transcribed text data is passed to the generative AI model.

[0060] The "means of systematizing feedback content for the role of sales training professionals" refers to a method or system for organizing and systematizing the feedback provided by sales training experts based on the evaluation results obtained from the generative AI model.

[0061] A "means for simultaneously training sales representatives and crew" is a method or system for simultaneously educating and training both sales representatives and new crew members.

[0062] "Means for storing recorded data on a cloud server" refers to the technology and process for storing acquired voice data on a cloud server via the Internet.

[0063] "Means for analyzing stored voice data" refers to technology or a system for analyzing voice data stored on a cloud server and generating evaluations and feedback.

[0064] "Means for providing evaluation results from a generative AI model to a user" refers to a method or system for communicating the evaluation results generated by the generative AI model to a user.

[0065] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0066] The present invention relates to a system for recording the role-playing of a new crew member, analyzing the content of the recording, and providing feedback. A specific embodiment of this system will be described below.

[0067] System Configuration

[0068] This system is configured using the following hardware and software.

[0069] 1. Hardware:

[0070] Microphone: Used to capture audio for roleplaying of new crew members.

[0071] Smartphone: Use the built-in recording function to capture audio data.

[0072] 2. Software:

[0073] Recording application: Record audio data using a recording application installed on your smartphone.

[0074] Data storage system: The acquired voice data is stored on a cloud server.

[0075] Generative AI model: Analyzes captured audio data and generates role-playing evaluations and feedback.

[0076] System Operation

[0077] Acquiring audio data:

[0078] Before starting role-playing, the user launches the recording application on the smartphone. The device (smartphone) picks up the role-playing audio through the microphone and starts recording. The user performs role-playing and taps the "stop recording" button when finished.

[0079] Audio data storage:

[0080] After the device finishes recording, it uploads the acquired voice data to the cloud server, and the server stores the uploaded voice data in the data storage system.

[0081] Analysis of audio data:

[0082] The server inputs the saved voice data into the generative AI model for analysis. The generative AI model analyzes the voice data and evaluates it on the evaluation items of role-playing (clarity of pronunciation, flow of speech, naturalness of dialogue, and overall evaluation).

[0083] Generate feedback:

[0084] The server receives the evaluation results from the generative AI model and generates feedback to provide to the user. The server sends the generated feedback to the user's device. The user checks the feedback on the device and understands what needs to be improved.

[0085] Examples

[0086] As a concrete example, the following prompt sentence is input into the generative AI model.

[0087] Example prompt:

[0088] "Please analyze the audio data of the new crew member's role-playing and rate it on the following items: 1. Clarity of pronunciation 2. Flow of speech 3. Naturalness of dialogue 4. Overall rating."

[0089] By inputting this prompt sentence into the generative AI model, the model analyzes the voice data and outputs an evaluation for each item. For example, the generative AI model generates feedback like the following:

[0090] Feedback example:

[0091] "Clarity of pronunciation: 8 / 10, flow of speech: 7 / 10, naturalness of dialogue: 9 / 10, overall rating: 8 / 10. Pronunciation was generally clear, but some words were hard to hear. Flow of speech was good, but could have been smoother. Naturalness of dialogue was very high, and overall it was good role-playing."

[0092] In this way, a system is realized in which the server, terminals, and users cooperate to record and evaluate the role-playing of new crew members.

[0093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0094] Step 1: Prepare to record

[0095] Before starting role-playing, the user launches the recording application on the smartphone. The device displays a pop-up message notifying the completion of the application launch. The input is the user's operation, and the output is the notification that the recording application has been launched.

[0096] Step 2: Acquire audio data

[0097] The user taps the "Start Recording" button on the recording application. The device picks up the role-playing voice in real time through the microphone and starts recording. The user performs the role-playing and taps the "Stop Recording" button when finished. The input is the user's voice, and the output is the recorded voice data.

[0098] Step 3: Save the audio data

[0099] The device temporarily stores the audio data after recording in its internal storage. The device then uploads the stored audio data to a cloud server. The server receives the uploaded audio data and stores it in a data storage system. The input is the recorded audio data, and the output is the audio data stored in the cloud server.

[0100] Step 4: Analyze the audio data

[0101] The server inputs the stored voice data into the generative AI model. The generative AI model analyzes the voice data and evaluates it on the evaluation items of role-playing (clarity of pronunciation, flow of story, naturalness of dialogue, and overall evaluation). The input is the voice data stored on the cloud server, and the output is the evaluation result.

[0102] Step 5: Generate feedback

[0103] The server receives the evaluation results from the generative AI model and generates feedback to provide to the user. The server sends the generated feedback to the user's device. The user checks the feedback on the device and understands future improvements. The input is the evaluation results from the generative AI model, and the output is the feedback provided to the user.

[0104] (Application example 1)

[0105] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] It is necessary to provide effective feedback and train new crew members quickly and efficiently in role-playing. However, with conventional methods, analyzing the recorded content and providing feedback is time-consuming and difficult, making it difficult to train efficiently.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0108] In this invention, the server includes a means for recording the role-playing of newcomers, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for simultaneously training sales representatives and crew members, a means for providing feedback from the generative AI model to new crew members, and a means for pointing out areas for improvement based on the feedback. This allows for quick and effective feedback to be provided to the role-playing of new crew members, enabling efficient training.

[0109] "Means for recording the role-playing content of new crew members" refers to a device or function for acquiring audio of the role-playing performed by new crew members.

[0110] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[0111] "Means for inputting the transcribed content into the generative AI model" refers to means for sending transcribed text data to the generative AI model for analysis and feedback.

[0112] "Means of systematizing feedback content from the perspective of a sales training professional" refers to a means of organizing the feedback obtained from the generative AI model and systematizing it from the perspective of a sales training professional.

[0113] "Means for training sales representatives and crew members simultaneously" refers to means for training both sales representatives and new crew members simultaneously.

[0114] "Means for providing feedback from the generative AI model to new crew members" refers to means for providing feedback to new crew members based on the results of analysis by the generative AI model.

[0115] "Means for pointing out areas for improvement based on feedback" refers to means for specifically pointing out areas for improvement for new crew members based on feedback from the generative AI model.

[0116] The system for implementing the present invention includes a series of processes for recording the role-playing content of a new crew member, transcribing the recorded content, inputting the content into a generative AI model, and providing feedback. A specific embodiment of the system will be described below.

[0117] Hardware and Software Configuration

[0118] The system uses the following hardware and software:

[0119] Hardware: Smartphone microphone

[0120] Software: Python, sounddevice library (audio recording), requests library (HTTP requests)

[0121] Data processing and calculation

[0122] 1. Recording:

[0123] The role-playing of new crew members is recorded in real time using a smartphone microphone.

[0124] The recorded audio data is obtained as a numpy array.

[0125] 2. Transcription:

[0126] The recorded voice data is converted into text information using voice recognition technology.

[0127] The transcribed text data is fed into a generative AI model.

[0128] 3. Analysis using generative AI models:

[0129] The transcribed text data is sent to a generative AI model for analysis.

[0130] The generative AI model analyzes the audio data and generates feedback.

[0131] 4. Providing Feedback:

[0132] The feedback obtained from the generative AI model is provided to new crew members.

[0133] Based on the feedback, we will provide specific areas for improvement for new crew members.

[0134] Examples

[0135] New crew members role-play customer service and record their voices. The recorded data is sent to a generative AI model, which provides feedback such as "your voice tone is too low" or "your greeting to customers is insufficient." This allows new crew members to identify specific areas for improvement and efficiently improve their skills.

[0136] Examples of prompt statements

[0137] Analyze the role-playing audio of the new crew member and provide feedback on the following:

[0138] 1. Tone of voice

[0139] 2. Speaking Speed

[0140] 3. Greeting customers

[0141] 4. Responding to Questions

[0142] In this way, training of new staff members in physical stores can be made more efficient and high quality customer service can be achieved.

[0143] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0144] Step 1:

[0145] The terminal (smartphone) records the role-playing content of the new crew member. The recording is done using the smartphone's microphone, and the audio data is obtained as a numpy array. The input is the new crew member's voice, and the output is the recorded audio data.

[0146] Step 2:

[0147] The device converts the recorded voice data into text information using voice recognition technology. Specifically, the voice recognition software analyzes the voice data and generates corresponding text data. The input is the voice data, and the output is the transcribed text data.

[0148] Step 3:

[0149] The terminal sends the transcribed text data to the generative AI model. Specifically, the terminal sends the text data as an HTTP request to the API of the generative AI model. The input is the transcribed text data, and the output is the request sent to the generative AI model.

[0150] Step 4:

[0151] The server (generative AI model) analyzes the received text data and generates feedback. The generative AI model evaluates the text data based on the prompt sentence and generates specific feedback. The input is the text data and the output is the feedback content.

[0152] Step 5:

[0153] The server sends the generated feedback back to the terminal. Specifically, it sends the feedback content to the terminal as an HTTP response. The input is the feedback content from the generative AI model, and the output is the feedback sent to the terminal.

[0154] Step 6:

[0155] The terminal provides the received feedback to the new crew member. Specifically, the feedback content is displayed on a display screen so that the new crew member can check it. The input is the feedback content from the server, and the output is the feedback provided to the new crew member.

[0156] Step 7:

[0157] The user (new crew member) identifies areas for improvement based on the provided feedback and reflects them in the next role-playing. Specifically, the user checks the feedback and performs the role-playing while keeping in mind the areas for improvement pointed out. The input is the feedback, and the output is the improved role-playing.

[0158] Example 2

[0159] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] In conventional systems that convert voice data into text information, the process from acquiring voice data to converting it into text information and then using it was complicated, making it difficult to carry out efficiently. There were also issues with the accuracy of converting voice data into text information using voice recognition technology, and with the management and use of the converted data. This caused problems with the time and effort required when users tried to use it as a prompt to input into a generative AI model.

[0161] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice data, a means for transmitting the acquired voice data to the server, a means for converting the voice data into character information, a means for storing the converted character information in a database, and a means for inputting the stored character information into a generative AI model. This makes it possible to efficiently perform the process from acquiring voice data to converting it into character information and using it.

[0162] "Audio data" refers to data in which audio is recorded in digital format.

[0163] "Means of acquisition" refers to devices and software for collecting voice data.

[0164] "Transmitting means" refers to a communication means for transferring the acquired voice data to a server.

[0165] "Means for converting into text information" refers to technology that analyzes voice data and generates corresponding text data.

[0166] "Means of storing in a database" refers to a system for systematically storing the converted character information.

[0167] "Means of input to the generative AI model" refers to the interfaces or processes used to provide the stored textual information to the AI ​​model.

[0168] "Means of recording in real time" refers to devices or software for instantly recording audio as digital data.

[0169] "Speech recognition technology" refers to technology for analyzing voice data and generating corresponding text information.

[0170] The present invention is a system for efficiently converting voice data into text information and inputting the text information into a generative AI model. A specific embodiment of the system will be described below.

[0171] First, a user uses a terminal for acquiring voice data. The terminal includes a voice input device such as a smartphone and a microphone. The user uses these devices to record the voice data. For example, when recording the contents of a meeting, the user can use a recording app on the smartphone.

[0172] The device then transmits the captured voice data to a server over the Internet, using HTTP requests to upload the voice data to the server, which then runs software to process the received voice data.

[0173] The server converts the received voice data into text information. This conversion is performed using voice recognition technology. Specifically, a voice recognition API or voice recognition service can be used. For example, Google's voice recognition API or IBM's Watson can be used. The server sends the voice data to these services and receives the text information in return.

[0174] The server then stores the converted text in a database, along with other metadata such as user ID and timestamp, to make it easier for users to search for the data later.

[0175] Finally, the user uses the saved text information as a prompt to be input to the generative AI model. The user opens a web browser and logs into the system's web interface. The user accesses the "Meeting Records" page and checks the saved text information. If necessary, the user copies the text information and uses it as a prompt to be input to the generative AI model.

[0176] As a concrete example, consider a case where a user wants to record the audio of a meeting and save the content as text information. The user uses a smartphone to record the audio of the meeting and sends the audio data to a server. The server uses a speech recognition API to convert the audio data into text information and stores the text information in a database. Later, the user can use the text information as a prompt sentence to input into a generative AI model.

[0177] An example of a prompt might be:

[0178] "Please summarize the following meeting: 'In today's meeting, we discussed the launch date of the new product. The marketing team proposed to set the launch date on the 15th of next month, but the development team said that it is not ready yet.'"

[0179] In this way, users can efficiently convert voice data into textual information that can be used in generative AI models.

[0180] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0181] Step 1:

[0182] A user obtains audio data using a smartphone or microphone. Specifically, the user opens a recording app on the smartphone and taps the "Start Recording" button to record the audio of the meeting. When recording is finished, the user taps the "Stop Recording" button to save the audio data. The input is audio, and the output is digital audio data.

[0183] Step 2:

[0184] The terminal transmits the acquired voice data to the server. Specifically, the user selects the saved voice data from the recording app and taps the "Send" button. This uploads the voice data to the server via an HTTP request. The input is the digital voice data, and the output is the voice data transmitted to the server.

[0185] Step 3:

[0186] The server converts the received voice data into text information. Specifically, the server sends the voice data to a voice recognition API, which analyzes the voice data and generates corresponding text information. For example, the server sends voice data to a voice recognition service and receives the returned text information. The input is voice data and the output is text information.

[0187] Step 4:

[0188] The server saves the converted text information in a database. Specifically, the server inserts the received text information into the "meeting records" table in the database. At this time, metadata such as the user ID and timestamp are also saved. The input is text information and metadata, and the output is the text information saved in the database.

[0189] Step 5:

[0190] The user uses the saved text information as a prompt sentence to be input to the generative AI model. Specifically, the user opens a web browser and logs into the system's web interface. The user accesses the "Meeting Records" page and checks the saved text information. If necessary, the user copies this text information and uses it as a prompt sentence to be input to the generative AI model. The input is the text information obtained from the database, and the output is the prompt sentence to be input to the generative AI model.

[0191] (Application example 2)

[0192] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] In conventional food delivery services, when delivery personnel manually input order details from customers, input errors and time loss were an issue. In addition, delivery personnel had to use their hands to input the order details, which made it difficult to carry out work efficiently.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for acquiring voice data, a means for converting the acquired voice data into text information, and a means for registering the converted text information in the order system. This makes it possible to efficiently process the order details input by voice by the delivery person.

[0195] "Audio data" refers to data in which audio is recorded in digital format.

[0196] "Means for acquiring" refers to a device or method for collecting audio data.

[0197] "Text information" refers to information obtained by converting voice data into text.

[0198] "Means of converting" refers to technology or equipment for converting voice data into text information.

[0199] "Order System" refers to a system for managing and processing orders from customers.

[0200] "Means for registering" refers to a method or device for inputting and storing converted character information into the ordering system.

[0201] "Management Means" means the methods and devices used to organize, track and manage orders.

[0202] A "delivery person" refers to a person whose role is to deliver goods to customers.

[0203] "Mobile device" refers to an electronic device that can be carried by a delivery person.

[0204] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text information.

[0205] To implement this invention, you need a mobile terminal for acquiring voice data, voice recognition technology for converting the voice data into text information, and a means for registering the converted text information in the ordering system.

[0206] System Configuration

[0207] 1. Mobile terminal: A mobile terminal such as a smartphone or smart glasses is used by the delivery person. This allows the delivery person to input the order details from the customer by voice.

[0208] 2. Speech recognition technology: To convert voice data into text information, we use voice recognition services such as Google's speech recognition API and IBM's Watson, which allows us to convert voice data into text information with high accuracy.

[0209] 3. Ordering system: An ordering system is used to manage and process the converted text information. This system has the function of automatically registering and managing the order contents.

[0210] Program Processing

[0211] The server first receives voice data from the mobile device. It then converts the voice data into text using voice recognition technology. The converted text is automatically registered in the order system. This allows delivery personnel to input order details without using their hands, making work more efficient.

[0212] Examples

[0213] For example, if a delivery person voice-inputs "one pizza, two colas," the mobile device captures this voice data. Next, the voice data is converted into text information using voice recognition technology, and registered in the ordering system as "one pizza, two colas." This series of processes allows the delivery person to quickly and accurately register the order details in the system.

[0214] Examples of prompt statements

[0215] "Please convert the following audio data into text: 1 pizza, 2 cokes"

[0216] In this way, a series of processes can be realized, from converting voice data into text information and registering it in the order system. This system improves the work efficiency of delivery personnel and reduces the risk of input errors.

[0217] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0218] Step 1:

[0219] The user (delivery person) inputs voice data using a mobile terminal. Specifically, the delivery person speaks the order details into a smartphone or smart glasses. The input is recorded as voice data on the mobile terminal.

[0220] Step 2:

[0221] The terminal acquires the voice data. The acquired voice data is temporarily stored in the memory of the terminal. The input is the delivery person's voice data, and the output is the voice data stored in the terminal.

[0222] Step 3:

[0223] The terminal transmits the acquired voice data to the server. Specifically, the terminal uploads the voice data to the server via the Internet. The input is the voice data stored in the terminal, and the output is the voice data transmitted to the server.

[0224] Step 4:

[0225] The server receives the voice data and converts it into text information using voice recognition technology. Specifically, the server calls Google's voice recognition API or IBM's Watson to convert the voice data into text information. The input is the voice data sent to the server, and the output is the converted text information.

[0226] Step 5:

[0227] The server registers the converted text information in the ordering system. Specifically, the server saves the text information in the ordering system's database and registers it as the order details. The input is the converted text information, and the output is the order details registered in the ordering system.

[0228] Step 6:

[0229] The ordering system manages the registered order details. Specifically, the ordering system organizes the order details and sends notifications to delivery personnel and customers as necessary. The input is the order details registered in the ordering system, and the output is the managed order information.

[0230] Step 7:

[0231] The user (delivery person) receives a notification from the ordering system and confirms the order details. Specifically, the delivery person receives a notification from the ordering system through a mobile terminal and confirms the order details. The input is the notification from the ordering system, and the output is the order details confirmed by the delivery person.

[0232] Example 3

[0233] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] In conventional sales training systems, it was difficult to effectively provide feedback and systematize the role-playing content of new recruits. In addition, the quality of the generated conversations and the accuracy of the feedback were low, making it inefficient to simultaneously train sales representatives and crew members. This limited the effectiveness of the training and made it difficult to improve practical skills.

[0235] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0236] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for receiving the generated conversation, a means for systematizing the feedback content of the generated conversation using text analysis and natural language processing technology, and a means for simultaneously training the sales representative and the crew. This improves the quality of the generated conversation, increases the accuracy of the feedback, and makes it possible to efficiently train the sales representative and the crew at the same time.

[0237] "Recording means" refers to a device or software that records the role-playing content of a newcomer as audio data.

[0238] A "transcription means" is a technology or device that converts recorded audio data into text data.

[0239] A "generative AI model" is an artificial intelligence algorithm or system that generates natural-sounding conversations based on input text information.

[0240] A "means for receiving the generated conversation" is a device or software that obtains the conversation data output from the generative AI model.

[0241] "Text analysis" is a technique or method for analyzing text data and understanding its meaning and structure.

[0242] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[0243] A "means for systematizing feedback content" is a technique or device for structuring and organizing feedback based on the generated conversation as a professional role in sales training.

[0244] A "means for simultaneously training salespeople and crew" is a system or method for effectively training salespeople and new crew members simultaneously using generated conversations and feedback.

[0245] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0246] This invention is a system that records the role-playing content of new recruits, transcribes the recording content, inputs it into a generative AI model, and organizes the generated conversation as feedback. A specific embodiment of this system will be described below.

[0247] Server Roles

[0248] The server receives character information input by the user. Specifically, the server receives an HTTP request and extracts character information from the request body. Next, the server inputs the received character information into a generative AI model. For example, GPT-4 (registered trademark) by OpenAI (registered trademark) can be used as this generative AI model. The server sends the character information to the generative AI model as an API request and receives the generated conversation. The generated conversation is analyzed using text analysis and natural language processing technology, and the feedback content is systematized. For this analysis, Python's NLTK library or SpaCy can be used.

[0249] Terminal Roles

[0250] The terminal transmits the character information input by the user to the server. Specifically, the terminal transmits the character information input by the user to the server as an HTTP request. After receiving the generated conversation from the server, the terminal displays the conversation to the user. Furthermore, the terminal provides an interface for the user to input feedback. This allows the user to input feedback for the generated conversation.

[0251] User Roles

[0252] The user inputs text information using the terminal. Specifically, the user inputs text information using a keyboard or a touch screen of the terminal. The user checks the generated conversation on the terminal and inputs feedback as necessary. The user inputs feedback using a feedback input interface of the terminal and checks that the content is organized.

[0253] Examples

[0254] For example, in a sales training scenario, a user (sales trainee) types into the terminal, "Please explain the features of the new product." The terminal sends this text information to the server, which then sends the text information to GPT-4 as an API request. GPT-4 generates a dialogue such as, "The features of the new product are that it is lightweight, easy to carry, and has a long battery life." The server receives the generated dialogue and systematizes the feedback content using text analysis and natural language processing techniques. The terminal displays the generated dialogue and feedback content to the user, who then checks it and enters feedback.

[0255] Examples of prompt statements

[0256] Prompt: "Describe the features of your new product."

[0257] When this prompt sentence is input to GPT-4, the AI ​​generates a natural conversation and provides useful feedback for the sales training scenario. The flow of the specific process in Example 3 is explained with reference to FIG.

[0258] Program processing flow

[0259] Step 1: Enter text information

[0260] Subject: User

[0261] The user uses the terminal to input text information. Specifically, the user uses the keyboard or touch screen of the terminal to input text information such as "Please explain the features of the new product." This input text information becomes the input data for the next step.

[0262] Step 2: Send text information

[0263] Subject: Terminal

[0264] The terminal sends the character information entered by the user to the server. Specifically, the terminal generates an HTTP request, includes the character information in the request body, and sends it to the server. The input of this step is the character information entered by the user, and the output is the HTTP request sent to the server.

[0265] Step 3: Receiving text information

[0266] Subject: Server

[0267] The server receives the HTTP request sent from the terminal and extracts text information from the request body. Specifically, the server analyzes the HTTP request and obtains the text information. The input of this step is the HTTP request from the terminal, and the output is the extracted text information.

[0268] Step 4: Inputting text information into the AI ​​model

[0269] Subject: Server

[0270] The server inputs the extracted character information into the generative AI model. Specifically, the server generates an API request to the generative AI model (e.g., GPT-4) and sends the character information. The input of this step is the extracted character information, and the output is an API request to the generative AI model.

[0271] Step 5: Receive the generated conversation

[0272] Subject: Server

[0273] The server receives the generated conversation from the generative AI model. Specifically, the server receives the API response from the generative AI model and extracts the generated conversation. The input of this step is the API response from the generative AI model, and the output is the generated conversation.

[0274] Step 6: Organize your feedback

[0275] Subject: Server

[0276] The server organizes the feedback content of the generated conversation using text analysis and natural language processing techniques. Specifically, the server analyzes the generated conversation and organizes the feedback content using Python's NLTK library and SpaCy. The input of this step is the generated conversation, and the output is the organized feedback content.

[0277] Step 7: Receive the generated conversation

[0278] Subject: Terminal

[0279] The terminal receives the generated conversation from the server. Specifically, the terminal receives the HTTP response from the server and extracts the generated conversation. The input of this step is the HTTP response from the server, and the output is the generated conversation.

[0280] Step 8: View the generated conversation

[0281] Subject: Terminal

[0282] The terminal displays the generated conversation to the user. Specifically, the terminal displays the generated conversation on a user interface. The input of this step is the generated conversation, and the output is the conversation displayed to the user.

[0283] Step 9: Provide feedback

[0284] Subject: User

[0285] The user inputs feedback for the generated conversation. Specifically, the user inputs feedback through a feedback input interface of the terminal. The input of this step is the generated conversation, and the output is the feedback input by the user.

[0286] Step 10: Review the feedback

[0287] Subject: User

[0288] The user confirms that the feedback content is organized. Specifically, the user confirms the result of the feedback content organization displayed on the terminal screen. The input of this step is the organized feedback content, and the output is the confirmed feedback content.

[0289] (Application example 3)

[0290] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Conventional sales training systems lacked a means to effectively provide feedback on the role-playing content of new employees, which led to slow improvement in sales skills. In addition, because the feedback content was not systematized, it was difficult to provide appropriate guidance to individual sales representatives. Furthermore, there was a need for an efficient training method that utilizes smartphones.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0293] In this invention, the server includes a means for recording the role-playing content of new recruits, a means for transcribing the recorded content, a means for inputting the transcribed content into a generative AI, a means for systematizing the feedback content as a professional role of sales training, a means for simultaneously training sales representatives and crew members, a means for analyzing the generated conversation using text analysis and natural language processing technology and providing feedback, and a means including an application to be installed on a smartphone. This makes it possible to improve the efficiency of sales training and provide effective feedback.

[0294] "Means for recording the role-playing content of new recruits" refers to devices or software for recording the audio of the role-playing performed by new recruits.

[0295] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[0296] "Means of inputting the transcribed content into the generative AI" refers to the interface or process for inputting the transcribed text data into the chat generative AI.

[0297] "Means of systematizing feedback content as a sales training professional" refers to techniques and methods for analyzing the generated conversation content from the perspective of a sales training professional and structuring the feedback.

[0298] A "means of simultaneous salesperson and crew development" is a system or method by which salespeople and new crew members learn and improve their skills at the same time.

[0299] "Means for analyzing the generated conversation using text analysis and natural language processing technology and providing feedback" refers to technologies and methods for evaluating the content of the generated conversation using text analysis and natural language processing technology and providing feedback based on the results.

[0300] "Means including an application installed on a smartphone" refers to a system or software including an application that runs on a smartphone.

[0301] A system for implementing this invention has the following configuration. First, a smartphone or a dedicated recording device is used as a means for recording the role-playing content of the new recruit. The recorded voice data is converted into text information using voice recognition technology. Google Speech-to-Text API, IBM Watson Speech to Text, etc. can be used for this voice recognition technology.

[0302] Next, the transcribed text data is input to a chat generation AI. OpenAI's GPT-3 (registered trademark) is used as this chat generation AI. GPT-3 can generate natural conversations based on the input text data.

[0303] The generated conversation content is analyzed using text analysis and natural language processing technology. Specifically, libraries such as TextBlob and NLTK (Natural Language Toolkit) are used to evaluate the positivity and subjectivity of the conversation. Based on the results of this analysis, feedback is provided from the perspective of sales training professionals.

[0304] This feedback is provided through an application installed on a smartphone. The application provides an interface for salespeople and new crew members to simultaneously learn and improve their skills. For example, it displays the conversations generated and their feedback, and allows users to evaluate their own performance.

[0305] As a concrete example, if a salesperson were simulating a conversation with a customer looking for a new smartphone, they might use the following prompt:

[0306] "Customer: Hello, I'm looking for a new smartphone today. Which model would you recommend?"

[0307] If we feed this prompt into a generative AI model, it might generate the following dialogue:

[0308] “Customer: Hello, I’m looking for a new smartphone today. Which model would you recommend?

[0309] Staff: Hello, customer. I understand you are looking for a new smartphone. We can recommend some models based on your needs. For example, the latest model is very popular and has a great camera. We can also recommend other high-performance models. What features do you care about?

[0310] The conversation is analyzed by TextBlob and feedback is provided. For example, feedback such as "Positiveness: 0.5, Subjectivity: 0.3" can be obtained. Based on this feedback, sales staff can improve their responses.

[0311] As described above, the present invention realizes efficient sales training and effective feedback provision by combining the means of recording, transcription, chat generation, text analysis, and feedback provision.

[0312] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0313] Step 1:

[0314] A user starts role-playing using a smartphone.

[0315] Input: User's voice

[0316] Output: Recorded audio data

[0317] Specific operation: The user activates the recording function of the smartphone and starts role-playing. The smartphone records the user's voice.

[0318] Step 2:

[0319] The server receives the recorded voice data and converts it into text information using voice recognition technology.

[0320] Input: Recorded audio data

[0321] Output: Transcribed text data

[0322] Specific operation: The server uses the Google Speech-to-Text API or IBM Watson Speech to Text to convert the voice data into text data.

[0323] Step 3:

[0324] The server inputs the transcribed text data into a chat generation AI, which generates natural conversation.

[0325] Input: Transcribed text data

[0326] Output: Generated conversation text

[0327] How it works: The server uses OpenAI's GPT-3 to generate natural-sounding conversations based on input text data.

[0328] Step 4:

[0329] The server analyzes the generated conversation text using text analysis and natural language processing techniques and provides feedback.

[0330] Input: Generated conversation text

[0331] Output: Feedback data (positivity, subjectivity, etc.)

[0332] Specific operation: The server uses TextBlob and NLTK to analyze the generated conversation text and evaluate its positivity and subjectivity.

[0333] Step 5:

[0334] The server sends the feedback data to the smartphone and displays it to the user.

[0335] Input: Feedback data

[0336] Output: Feedback information displayed to the user

[0337] Specific operation: The server sends the feedback data to the smartphone, and the smartphone application displays the feedback information to the user.

[0338] Step 6:

[0339] Users can view feedback and use it to improve their sales skills.

[0340] Input: Feedback information displayed to the user

[0341] Output: Understanding users and improving their behavior

[0342] Specific Actions: Users review the feedback information displayed on their smartphone screen, evaluate their sales skills, and identify areas for improvement.

[0343] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0344] "Example 1"

[0345] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system records the role-playing of a new crew member and transcribes the recorded content. The transcribed content is input to a generative AI, which organizes the feedback content as a professional role in sales training. Furthermore, this system includes an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from the recorded content and reflects the results in the feedback content. Specifically, the emotion is estimated from the user's tone of voice, speaking style, choice of words, etc., and the information is used in the feedback. This enables more specific and useful feedback that also takes into account the emotional reactions of the new crew member.

[0346] "Example 2"

[0347] The emotion engine also optimizes the training of salespeople and crew members based on the results of emotion analysis. Specifically, it analyzes how new crew members emotionally react to certain scenarios and how those reactions affect sales performance, and adjusts the training program accordingly. For example, if it senses that new crew members are anxious about tough questions from customers, it can provide training to strengthen their ability to handle such scenarios.

[0348] "Example 3"

[0349] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system records the role-playing of a new crew member and transcribes the recorded content. The transcribed content is input to a generative AI, which organizes the feedback content as a professional role in sales training. Furthermore, this system includes an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from the recorded content and reflects the results in the feedback content. Specifically, the emotion is estimated from the user's tone of voice, speaking style, choice of words, etc., and the information is used in the feedback. This enables more specific and useful feedback that also takes into account the emotional reactions of the new crew member.

[0350] The process flow of each embodiment will be described below.

[0351] "Example 1"

[0352] Step 1: Have new crew members role-play and record the session.

[0353] Step 2: Transcribe the recorded session.

[0354] Step 3: Input the transcribed content into the generative AI.

[0355] Step 4: The generative AI generates the feedback content.

[0356] Step 5: The emotion engine analyzes the user's emotions from the recording.

[0357] Step 6: The emotion engine applies the analysis results to the feedback content.

[0358] "Example 2"

[0359] Step 1: The emotion engine analyzes the user's emotions from the recording.

[0360] Step 2: The sentiment engine uses the results of sentiment analysis to optimize the training of sales representatives and crew.

[0361] Step 3: The emotion engine analyzes how the new crew member will emotionally react to a particular scenario.

[0362] Step 4: The emotion engine analyzes how that response impacts sales outcomes.

[0363] Step 5: The emotion engine adjusts the training program based on the analysis results.

[0364] Example 1

[0365] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0366] In the past, role-playing for new crew members required the manual process of transcribing recordings and providing feedback, which was time-consuming and labor-intensive. In addition, the feedback did not take into account emotional reactions, limiting the improvement of response skills in actual work. Furthermore, there was a lack of effective means to simultaneously train salespeople and crew members.

[0367] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the newcomer, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for performing emotion analysis using an emotion engine that recognizes the user's emotions, a means for reflecting the emotion analysis result in the feedback content, and a means for simultaneously training the salesperson and the crew. This makes it possible to automate the process from recording to providing feedback and provide specific and useful feedback that takes into account emotional reactions. In addition, it is possible to effectively train the salesperson and the crew at the same time.

[0368] "Means for recording the role-playing content of new crew members" refers to devices or functions for recording the audio of the role-playing performed by new crew members.

[0369] A "means of transcribing the recorded content" refers to technology or software for converting recorded audio data into text data.

[0370] "Means for inputting the transcribed content into the generative AI model" refers to the process of sending the transcribed text data to the generative AI model to generate feedback content.

[0371] "A means of systematizing feedback content for the role of a sales training professional" refers to a method that uses a generative AI model to structure and provide feedback like that given by a sales training professional.

[0372] The "means for performing emotion analysis using an emotion engine that recognizes the user's emotions" refers to software or algorithms for analyzing emotions from a user's voice or text.

[0373] The "means for reflecting the emotion analysis result in the feedback content" is a method for incorporating the emotion analysis result obtained by the emotion engine into the generated feedback content.

[0374] A "means of simultaneously developing salespeople and crew" is an educational program or system that allows salespeople and new crew members to learn and grow at the same time.

[0375] This invention is a system that effectively records the role-playing of new crew members, transcribes the contents, and provides feedback. Furthermore, by analyzing the user's emotions and reflecting the results in the feedback, it is possible to provide more specific and useful feedback.

[0376] Hardware and Software Used

[0377] Recording device

[0378] When the user role-plays a new crew member, the user can obtain voice data using a microphone or the recording function of the smartphone. For example, the user can use a voice memo app on the smartphone or a dedicated recording device.

[0379] Speech Recognition Software

[0380] The server uses speech recognition software to transcribe the captured audio data, specifically using a speech recognition service such as the Google Cloud Speech-to-Text API. This process converts the audio data into text data.

[0381] Generative AI Models

[0382] The server inputs the transcribed text data into a generative AI model, such as OpenAI's GPT-3, which then organizes the feedback into a professional role in sales training.

[0383] Sentiment Analysis Tools

[0384] The server uses an emotion engine to analyze the user's emotions from the recordings, using emotion analysis tools such as IBM Watson Tone Analyzer, which infers emotions from the user's tone of voice, speaking style, and choice of words.

[0385] Examples

[0386] New crew members role-play customer service and record the content on their smartphones. The recorded voice data is sent to a server and transcribed using Google Cloud Speech-to-Text API. The transcribed text data is then input into GPT-3, which generates feedback content as a professional role in sales training. In addition, sentiment analysis is performed using IBM Watson Tone Analyzer, and the results are reflected in the feedback.

[0387] Examples of prompt statements

[0388] "A new crew member has role-played interacting with a customer. Please transcribe the audio data below and generate feedback. Also, please perform sentiment analysis and incorporate the results into your feedback."

[0389] In this way, users can get specific, useful feedback on their role-playing of new crew members that also takes into account their emotional reactions.

[0390] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0391] Step 1:

[0392] When a user role-plays a new crew member, the user acquires voice data using a microphone or the recording function of the smartphone. Specifically, the user opens a voice memo app on the smartphone and presses the "Start Recording" button to record the role-playing. When the recording is finished, the user presses the "Stop Recording" button to save the voice data. The input is the voice of the role-playing, and the output is the recorded voice data.

[0393] Step 2:

[0394] The terminal uploads the recorded voice data to the server. The user performs this operation by selecting the voice file through a dedicated application or a web interface and pressing the "upload" button. The input is the recorded voice data, and the output is the voice data uploaded to the server.

[0395] Step 3:

[0396] The server receives the uploaded voice data and transcribes it using a speech recognition service such as Google Cloud Speech-to-Text API. The server sends the voice data to the API and stores the returned text data. The input is the uploaded voice data, and the output is the transcribed text data.

[0397] Step 4:

[0398] The server inputs the transcribed text data into a generative AI model (e.g., OpenAI's GPT-3). The server sends the text data to the generative AI model as a prompt sentence to generate feedback content. The input is the transcribed text data, and the output is the generated feedback content.

[0399] Step 5:

[0400] The server uses a sentiment analysis tool such as IBM Watson Tone Analyzer to analyze user sentiment from the transcribed text data. The server sends the text data to the sentiment analysis tool and stores the returned sentiment analysis results. The input is the transcribed text data, and the output is the sentiment analysis results.

[0401] Step 6:

[0402] The server integrates the feedback content obtained from the generative AI model with the sentiment analysis results. The server reflects the sentiment analysis results in the feedback content to generate the final feedback. The input is the generated feedback content and the sentiment analysis results, and the output is the integrated final feedback.

[0403] Step 7:

[0404] The server provides the final feedback to the user, who can review the feedback through a dedicated application or a web interface. The input is the integrated final feedback, and the output is the feedback provided to the user.

[0405] (Application example 1)

[0406] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] In the conventional training system for new crew members, the content of role-playing was recorded and transcribed, making it difficult to analyze emotions and improve the quality of feedback. In addition, the feedback was one-sided and it was not possible to provide specific advice that took emotional reactions into account. This made it difficult to effectively support the growth of new crew members.

[0408] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the new employee, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for analyzing the user's emotions using an emotion engine, a means for reflecting the results of the emotion analysis in the feedback content, and a means for simultaneously training the sales representative and the crew. This makes it possible to provide specific and useful feedback that takes into account emotional reactions.

[0409] A "newcomer" refers to an employee who has been newly hired by a company or organization.

[0410] "Role playing" refers to a training method in which actual tasks or situations are simulated by playing specific roles.

[0411] "Recording" refers to the act of saving sound on a recording medium.

[0412] "Transcription" refers to the process of converting audio data into text data.

[0413] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or data.

[0414] "Feedback" refers to the act of providing evaluation and suggestions for improvement for specific actions or results.

[0415] An "emotion engine" refers to a system that analyzes emotions from voice and text and outputs the results.

[0416] "Sales training" refers to the training and education provided to sales representatives to acquire the necessary knowledge and skills.

[0417] A "professional role" refers to a role played by someone with specialized knowledge or experience.

[0418] "Systematization" refers to organizing and structuring information and data.

[0419] "Development" refers to education and training to improve specific skills and abilities.

[0420] A system for implementing this invention has the following configuration. First, a user performs role-playing using a smartphone. The smartphone's microphone is used to record the voice of the role-playing. The recorded voice is saved using a recording application in the smartphone (e.g., the MediaRecorder class of ANDROID (registered trademark)).

[0421] Next, the recorded voice file is transcribed using a speech recognition API (e.g., Google Cloud Speech-to-Text API). The transcribed text is input to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions. The emotion engine estimates emotions from the tone of voice, speaking style, and choice of words, and outputs the results.

[0422] The transcribed text and the results of the sentiment analysis are then input into a generative AI model (e.g., OpenAI GPT-3), which uses this information to generate feedback that is then displayed in the smartphone application UI and provided to the user.

[0423] As a concrete example, by inputting the following prompt sentence into the generative AI model, feedback reflecting the results of sentiment analysis can be obtained.

[0424] Prompt statement:

[0425] Below is the text of a role-playing session for a new crew member. Please provide your feedback on this, taking into account the results of the sentiment analysis.

[0426] Role-playing content:

[0427] "Hello, welcome. What item would you like to purchase today?"

[0428] Sentiment analysis results:

[0429] Tone: Friendly

[0430] Emotion: Joy

[0431] feedback:

[0432] By inputting this prompt sentence into the generative AI model, feedback such as "Your greeting was very friendly and left a good impression on the customer. Next time, it would be even better if you added a more specific question." In this way, new crew members can effectively train in physical stores.

[0433] The system makes it possible to provide specific, useful feedback that takes emotional responses into account, effectively supporting the development of new crew members.

[0434] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0435] Step 1:

[0436] The user uses a smartphone to perform role-playing. The smartphone's microphone is used to record the audio of the role-playing. A recording application (e.g. Android (registered trademark) MediaRecorder class) is launched, and recording begins when the user presses the record button. The recorded audio is saved in the smartphone as an audio file.

[0437] Input: User's voice

[0438] Output: Recorded audio file

[0439] Step 2:

[0440] The device sends the recorded audio file to a speech recognition API (e.g., Google Cloud Speech-to-Text API). The speech recognition API converts the audio data into text data and returns the transcribed text. The device receives this text data.

[0441] Input: Recorded audio file

[0442] Output: Transcribed text data

[0443] Step 3:

[0444] The terminal inputs the transcribed text data into an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the user's emotion from the text data and returns the emotion analysis result. The terminal receives this emotion analysis result.

[0445] Input: Transcribed text data

[0446] Output: Sentiment analysis results

[0447] Step 4:

[0448] The device inputs the transcribed text data and the sentiment analysis results into a generative AI model (e.g., OpenAI GPT-3). The generative AI model generates feedback based on this information. The device receives this feedback.

[0449] Input: Transcribed text data, sentiment analysis results

[0450] Output: The generated feedback

[0451] Step 5:

[0452] The terminal displays the generated feedback on the application UI, and the user can check the feedback through the smartphone screen.

[0453] Input: Generated feedback

[0454] Output: Feedback displayed in the application UI

[0455] In this way, users can receive specific, useful feedback that takes into account their emotional reactions to the role-playing content.

[0456] Example 2

[0457] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0458] With traditional training methods for new crew members, it was difficult to provide optimal training based on individual emotional responses and sales results. In addition, the process of converting voice data into text information and analyzing emotions was often done manually and was inefficient. This limited the effectiveness of the training program and prevented crew members from maximizing their growth.

[0459] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0460] In this invention, the server includes a means for acquiring voice data, a means for transmitting the acquired voice data to the server, a means for converting the voice data into text information, a means for sentiment analysis of the text information, and a means for optimizing the training program based on the result of the sentiment analysis. This makes it possible to efficiently perform a series of steps from automatic acquisition of voice data, conversion to text information, sentiment analysis, and optimization of the training program.

[0461] "Audio data" refers to data in which audio is recorded in digital format.

[0462] A "server" is a computer system that processes and stores data over a network.

[0463] "Text information" refers to voice data converted into text format.

[0464] "Sentiment analysis" is the process of analyzing text data for emotions and emotional trends.

[0465] A "development program" is a training plan designed to improve the skills and knowledge of new crew members.

[0466] "Means for acquiring" refers to a device or method for collecting voice data.

[0467] The "transmitting means" refers to a device or method for transferring the acquired voice data to the server.

[0468] The "converting means" refers to a device or method for converting voice data into text information.

[0469] An "optimizing means" is a device or method for adjusting a training program based on the results of sentiment analysis.

[0470] The present invention is a system for acquiring voice data, converting it into text information, performing sentiment analysis, and optimizing a training program based on the results. A specific embodiment of this system will be described below.

[0471] First, the user uses a device such as a smartphone or tablet to record a conversation with a customer. The user starts the recording app on the device and presses the "Start Recording" button to record the conversation. When the conversation is over, the user presses the "Stop Recording" button to end the recording.

[0472] Next, the device transmits the recorded voice data to the server. The device selects the recording file and presses the "send" button to upload the voice data to the server via the Internet.

[0473] The server converts the received voice data into text using a voice recognition service such as Google's voice recognition API or IBM's Watson. The server sends the voice data to the voice recognition API, which analyzes the voice data and generates text information. The generated text information is then sent back to the server.

[0474] The server then inputs the text information into the emotion engine to perform emotion analysis. The server sends the text information to the emotion engine, which analyzes the text information and generates emotion analysis results. The results are sent back to the server.

[0475] Finally, the server optimizes the training program for new crew members based on the results of the sentiment analysis. The server evaluates the results of the sentiment analysis and analyzes how new crew members emotionally react to specific scenarios. For example, if it is analyzed that a new crew member feels anxious when asked a tough question by a customer, the server generates a training program to strengthen the ability to respond to that scenario.

[0476] As a concrete example, a user uses a smartphone to record a conversation with a customer, converts the voice data into text, and then performs sentiment analysis on the text. The following is an example of a prompt sentence that optimizes the training program for new crew members based on the results.

[0477] Example prompt:

[0478] "Record conversations with customers and convert the audio data into text. Then run the text through sentiment analysis and use the results to optimize your onboarding program."

[0479] In this way, the system can efficiently perform a series of processes from acquiring voice data to analyzing emotions and optimizing training programs.

[0480] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0481] Step 1:

[0482] Acquiring audio data

[0483] The user uses a device such as a smartphone or tablet to record a conversation with a customer. The user starts the recording app on the device and presses the "Start Recording" button to record the conversation. When the conversation is over, the user presses the "Stop Recording" button to end the recording.

[0484] Input: Customer conversation

[0485] Output: Recorded audio data

[0486] Step 2:

[0487] Sending voice data to the server

[0488] The device sends the recorded voice data to the server. The device selects the recording file and presses the "Send" button to upload the voice data to the server via the Internet.

[0489] Input: Recorded audio data

[0490] Output: Audio data uploaded to the server

[0491] Step 3:

[0492] Converting voice data into text

[0493] The server converts the received voice data into text using a voice recognition service such as Google's voice recognition API or IBM's Watson. The server sends the voice data to the voice recognition API, which analyzes the voice data and generates text information. The generated text information is then sent back to the server.

[0494] Input: Audio data uploaded to the server

[0495] Output: Text information

[0496] Step 4:

[0497] Conducting sentiment analysis

[0498] The server inputs the text information into the emotion engine and performs sentiment analysis. The server sends the text information to the emotion engine, which analyzes the text information and generates the result of the sentiment analysis. The result is sent back to the server.

[0499] Input: Text information

[0500] Output: Sentiment analysis results

[0501] Step 5:

[0502] Optimizing training programs

[0503] The server optimizes the training program for new crew members based on the results of the sentiment analysis. The server evaluates the results of the sentiment analysis and analyzes how new crew members react emotionally to specific scenarios. For example, if it is analyzed that a new crew member feels anxious when asked a tough question by a customer, a training program is generated to strengthen the ability to respond to that scenario.

[0504] Input: Sentiment analysis results

[0505] Output: Optimized breeding program

[0506] (Application example 2)

[0507] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0508] In conventional sales training systems, it was difficult to grasp the emotional state of new crew members in real time and provide appropriate feedback based on that. Therefore, when new crew members felt anxious or stressed, it was not possible to respond appropriately to their emotions, which led to a problem of reduced training effectiveness.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0510] In this invention, the server includes a means for recording the role-playing content of the new crew member, a means for transcribing the recorded content, a means for inputting the transcribed content into a sentiment analysis engine, a means for providing feedback based on the result of the sentiment analysis, and a means for simultaneously training the sales representative and the crew member. This makes it possible to grasp the emotional state of the new crew member in real time and provide appropriate feedback based on that.

[0511] The "means for recording the role-playing content of a new crew member" refers to a device or method for acquiring audio data of the role-playing performed by a new crew member.

[0512] A "means for transcribing recorded content" is a device or method for converting acquired voice data into text data.

[0513] "Means for inputting the transcribed content to a sentiment analysis engine" refers to an apparatus or method for transmitting text data to a sentiment analysis engine for analyzing the emotional state.

[0514] A "means for providing feedback based on the result of sentiment analysis" is a device or method for providing appropriate guidance or advice based on the emotional state result obtained from the sentiment analysis engine.

[0515] A "means for simultaneously developing sales representatives and crew" is a device or method for simultaneously educating or training both sales representatives and crew.

[0516] The system for implementing the present invention records the role-playing content of a new crew member, transcribes the recorded content, and performs sentiment analysis to provide appropriate feedback. A specific embodiment of this system will be described below.

[0517] 1. System Configuration

[0518] The system consists of the following main components:

[0519] Recording Equipment: Microphones for recording new crew members' role-playing in real time.

[0520] Speech recognition software: Speech recognition technology for converting recorded voice data into text information. Specifically, Google's speech recognition API and IBM's Watson are used.

[0521] Sentiment analysis engine: Software that receives text information as input and analyzes the emotional state. Specifically, it uses IBM Watson Tone Analyzer.

[0522] Feedback provision device: Displays and speakers for providing appropriate feedback to new crew members based on the results of sentiment analysis.

[0523] 2. Program Processing

[0524] The server first obtains the role-playing content of the new crew member using a recording device. Then, it converts the recorded content into text information using voice recognition software. After that, it inputs the text information into a sentiment analysis engine to analyze the emotional state. Finally, it provides appropriate feedback to the new crew member through a feedback providing device based on the result of the sentiment analysis.

[0525] 3. Hardware and Software Used

[0526] Hardware: Microphone, display, speakers

[0527] Software: Google speech recognition API, IBM Watson Tone Analyzer

[0528] 4. Specific Examples

[0529] For example, if a worker says, "This task is difficult," speech recognition will convert this into text information, and emotion analysis will detect "anxiety." As a result, the robot will provide feedback such as, "Don't worry, let's try a bit more slowly."

[0530] Examples of prompt statements

[0531] If a worker says, "This task is difficult," speech recognition converts this into text information, and emotion analysis detects "anxiety." As a result, the robot provides feedback such as, "Don't worry, let's try a little more slowly."

[0532] In this way, the system can grasp the emotional state of new crew members in real time and provide appropriate feedback based on that, which can improve the effectiveness of training new crew members.

[0533] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0534] Step 1:

[0535] The server uses a microphone to record the role-playing content of the new crew member in real time, the input is the voice data of the new crew member, and the output is the recorded voice file, which is used in subsequent processing steps.

[0536] Step 2:

[0537] The server inputs the recorded voice file into speech recognition software and converts it into text information. Specifically, it uses Google's speech recognition API. The input is the voice file, and the output is text information (text data). This text data is used for sentiment analysis.

[0538] Step 3:

[0539] The server inputs the text information into a sentiment analysis engine and analyzes the emotional state. Specifically, it uses IBM Watson Tone Analyzer. The input is text information, and the output is the result of sentiment analysis (emotional state data). This emotional state data is used to provide feedback.

[0540] Step 4:

[0541] The server generates feedback based on the result of the sentiment analysis and provides it to the new crew member through the feedback providing device. Specifically, the feedback is displayed or audibly conveyed using a display or speaker. The input is the result of the sentiment analysis, and the output is a feedback message. For example, specific advice such as "Don't worry, try doing it a little more slowly" is provided.

[0542] Step 5:

[0543] The server monitors the effectiveness of the feedback and adjusts the feedback content as necessary. The input is the new crew's reaction and performance data, and the output is the adjusted feedback message. This allows the effectiveness of training new crew members to be continuously improved.

[0544] Example 3

[0545] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0546] Conventional sales training systems only recorded and transcribed the role-playing of new crew members, but were unable to reflect the user's emotions in the generated conversations and feedback. As a result, the feedback was one-sided, making it difficult to provide specific and useful advice that took into account the user's emotional reactions. In addition, there was a lack of means to simultaneously train salespeople and crew members.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0548] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for generating a conversation from the transcribed content using natural language processing technology, a means for systematizing the feedback content based on the generated conversation, a means for analyzing the user's emotions and reflecting the results in the feedback, and a means for simultaneously training the sales representative and the crew. This enables specific and useful feedback that takes into account the user's emotional reactions, and effectively trains the sales representative and the crew at the same time.

[0549] "Means for recording the role-playing content of newcomers" refers to devices or software used to record the audio of role-playing performed by new crew members.

[0550] "Means of transcribing the recording" refers to speech recognition technology or software used to convert recorded audio data into text data.

[0551] "Means of generating conversation from transcribed content using natural language processing technology" refers to natural language processing technology or generative AI models that use transcribed text data as input to generate natural conversation.

[0552] "Means of systematizing the feedback content based on the generated conversation" refers to technology and software for analyzing the generated conversation data, extracting important points and areas for improvement, and organizing and structuring the feedback content.

[0553] "Means of analyzing the user's emotions and reflecting the results in the feedback" refers to emotion recognition technology and software that infers emotions from the user's tone of voice, speaking style, choice of words, etc., and incorporates that information into the feedback content.

[0554] "Salesperson and crew development tools" refers to educational programs and systems designed to enable salespeople and new crew members to learn and grow at the same time.

[0555] This invention is a system that records the role-playing of new crew members, transcribes the recordings, generates natural conversations using a generative AI model, and organizes the feedback content. Furthermore, it analyzes the user's emotions and reflects the results in the feedback, providing more specific and useful feedback.

[0556] First, the user role-plays as a new crew member dealing with customers. The user uses the microphone of the terminal to record the conversation. For example, a recording application on a smartphone can be used to record the conversation.

[0557] The server then receives the recorded voice data and transcribes it using voice recognition technology. Specifically, it uses voice recognition software such as the Google Speech-to-Text API. The transcribed text data is then stored on the server.

[0558] The server then retrieves the transcribed text data and inputs it into a generative AI model, such as OpenAI GPT-4. The AI ​​generates natural conversation based on the input text and returns the results to the server.

[0559] The generated conversation data is then analyzed by the server, which uses natural language processing libraries such as NLTK and spaCy to extract important points and areas for improvement. Based on this information, the feedback is organized into a professional role for sales training.

[0560] In addition, the server inputs the recorded content into emotion recognition software to analyze the user's emotions. Specifically, it uses "IBM Watson Tone Analyzer" and other software. The server infers emotions from the user's tone of voice, speaking style, and choice of words, and reflects this information in the feedback.

[0561] For example, a new crew member role-plays a customer interaction and records the content. The recorded audio is transcribed using the Google Speech-to-Text API. The transcribed text is input into OpenAI GPT-4 to generate natural conversation. The generated conversation is analyzed using NLTK and spaCy, and the feedback content is organized as a professional role in sales training.

[0562] In addition, the system uses IBM Watson Tone Analyzer to analyze the user's emotions and reflects the results in the feedback. For example, if the user is nervous, the feedback will include that information, providing specific advice.

[0563] An example of a prompt sentence might be:

[0564] "A new crew member has role-played a customer interaction. Please transcribe the recording below and feed it into your generative AI model to generate natural-sounding conversations. Also, please use the emotion engine to analyze the user's emotions and incorporate the results into your feedback."

[0565] In this way, the system records the role-playing of the new crew member, transcribes it, generates a conversation, analyzes emotions, and finally organizes the feedback content. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0566] Step 1:

[0567] The user role-plays as a new crew member dealing with customers. The conversation is recorded using the device's microphone. The recorded voice data is input and saved in the device.

[0568] Step 2:

[0569] The server receives the recorded voice data from the device. The server uses "voice recognition technology" to convert the voice data into text data. Specifically, it calls the "voice recognition API" and uses the voice data as input to output text data. The converted text data is stored on the server.

[0570] Step 3:

[0571] The server retrieves the stored text data. The server uses the "generative AI model" to generate natural conversation using the text data as input. Specifically, it calls the "generative AI model API" and outputs natural conversation data using the text data as input. The generated conversation data is stored on the server.

[0572] Step 4:

[0573] The server obtains the generated conversation data. The server uses "natural language processing technology" to analyze the conversation data and extract important points and areas for improvement. Specifically, it uses a "natural language processing library" to input the conversation data and output feedback content. The feedback content is stored on the server.

[0574] Step 5:

[0575] The server analyzes the recorded content using "emotion recognition technology." Specifically, it calls the "emotion recognition API" and uses the recorded data as input to estimate the user's emotions. The emotion analysis results are stored on the server.

[0576] Step 6:

[0577] The server obtains the emotion analysis results and reflects them in the feedback content. Specifically, the server modifies the feedback content based on the emotion analysis results and provides specific advice that takes into account the user's emotional response. The final feedback content is provided to the user.

[0578] (Application example 3)

[0579] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0580] In the conventional training system for new operators, feedback was limited to general content and did not fully take into account the feelings and specific problems of individual operators. In addition, there was a lack of specific training on the operation of factory robots, making it difficult to provide efficient training.

[0581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the newcomer, a means for transcribing the recorded content, a means for inputting the transcribed content to the generative AI, a means for systematizing the feedback content as a professional role in sales training, a means for analyzing the user's emotions using an emotion engine and reflecting the results in the feedback content, and a means for educating new factory robot operators. This enables more specific and useful feedback that takes into account the emotions and specific problems of each operator.

[0582] "Means for recording the role-playing content of new recruits" refers to devices or software for recording the audio of role-playing performed by new operators.

[0583] A "means for transcribing recorded content" is a device or software that uses voice recognition technology to convert recorded voice data into text data.

[0584] "Means of inputting the transcribed content into a generative AI" refers to devices or software that input the transcribed text data into a chat generative AI and generate feedback in a natural conversational format.

[0585] The "means for systematizing feedback content from the perspective of a sales training professional" refers to a device or software for systematizing the generated feedback from the perspective of a sales training professional and organizing it as educational content.

[0586] "Means for analyzing a user's emotions using an emotion engine and reflecting the results in the feedback content" refers to a device or software that uses emotion recognition technology to analyze emotions from a user's voice or text and reflect the results in the feedback.

[0587] The "means for training new factory robot operators" refers to devices and software for training new operators on how to operate factory robots.

[0588] The system for carrying out the present invention is for efficiently training new operators. The system has the following configuration.

[0589] First, a new operator role-plays how to operate a factory robot. The content of this role-play is recorded in real time using a recording device. The recorded voice data is converted into text data using voice recognition technology. For example, the "speech_recognition" library is used for this voice recognition technology.

[0590] The transcribed text data is then fed into a chat generation AI, such as the GPT-3 model, which generates feedback in a natural conversational format based on the input text data.

[0591] Furthermore, the emotion engine is used to analyze the user's emotions. For example, the "emotion_recognition" library is used for this emotion engine. The emotion engine analyzes emotions from the user's voice and text, and reflects the results in the feedback.

[0592] Finally, the generated feedback is organized into a professional sales training roll, which contains specific advice and improvements that will help educate new operators.

[0593] As a concrete example, consider a scenario in which a new operator has a question about operating a factory robot. By inputting the following prompt sentences into the generative AI model, feedback can be obtained in a natural, conversational format.

[0594] Example prompt:

[0595] A new operator has a question about how to operate a factory robot. Generate feedback in a natural conversational format based on the text below.

[0596] Text: "I don't know how to operate this robot. What should I do?"

[0597] The system allows for more specific and useful feedback that takes into account each operator's feelings and specific problems, improving the efficiency of training new operators and helping them quickly acquire skills in operating factory robots.

[0598] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0599] Step 1:

[0600] The user role-plays the operation of a factory robot. The terminal records the voice of this role-play in real time using a recording device. The input is the user's voice, and the output is the recorded voice data.

[0601] Step 2:

[0602] The server converts the recorded voice data into text data using speech recognition technology. The "speech_recognition" library is used for this speech recognition technology. The input is the recorded voice data, and the output is the transcribed text data.

[0603] Step 3:

[0604] The server inputs the transcribed text data into a chat generation AI. The GPT-3 model is used for this chat generation AI. The input is the transcribed text data, and the output is feedback in a natural conversational format.

[0605] Step 4:

[0606] The server analyzes the user's emotions using an emotion engine, which uses the "emotion_recognition" library. The input is the transcribed text data, and the output is the result of the emotion analysis.

[0607] Step 5:

[0608] The server combines the generated feedback with the results of sentiment analysis and organizes the feedback content as a professional role in sales training. The input is the natural conversational feedback and the results of sentiment analysis, and the output is the organized feedback content.

[0609] Step 6:

[0610] The terminal provides the user with structured feedback content, where the input is the structured feedback content and the output is specific advice or improvements for the user.

[0611] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0613] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0614] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0615] [Second embodiment]

[0616] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0617] .

[0618] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0619] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0620] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0621] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0622] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0623] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0624] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0625] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0626] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0627] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0628] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0629] "Example 1"

[0630] In one embodiment of the present invention, a device is provided for recording the role-playing of new crew members. This device obtains voice data of the role-playing using, for example, a microphone or a recording function of a smartphone.

[0631] "Example 2"

[0632] Next, the acquired voice data is converted into text information. This conversion is performed, for example, using voice recognition technology. Specifically, it is possible to use voice recognition services such as Google's voice recognition API or IBM's Watson.

[0633] "Example 3"

[0634] The converted text information is then input into a generative AI, which is an AI that can mimic human conversation and generate natural conversation based on the input text information.

[0635] "Example 4"

[0636] Finally, the feedback content of the generated conversation is systematized as a professional role in sales training. This systematization can be done, for example, using text analysis and natural language processing technology.

[0637] The process flow of each embodiment will be described below.

[0638] "Example 1"

[0639] Step 1: The new crew member begins role-playing. This role-playing is based on a sales scenario, in which the new crew member simulates interactions with a customer.

[0640] Step 2: Once the role-playing session has started, record it using a microphone or smartphone recording function. This recording is done throughout the role-playing session, capturing in detail the comments and reactions of the new crew member.

[0641] "Example 2"

[0642] Step 1: Once the recording is complete, the audio data is converted into text information. This conversion is done using voice recognition technology. Specifically, it uses voice recognition services such as Google's voice recognition API or IBM's Watson.

[0643] Step 2: The speech recognition service converts the speech data into text data and outputs the result, which is used in the following steps.

[0644] "Example 3"

[0645] Step 1: The text data output from the speech recognition service is input into a generative AI, which is an AI that can imitate human conversation.

[0646] Step 2: The generative AI generates natural conversations based on the input text data. This generated conversation is used as feedback to new crew members.

[0647] "Example 4"

[0648] Step 1: Systematize the feedback content generated by the generative AI as a professional role in sales training. This systematization is done using text analysis and natural language processing technology.

[0649] Step 2: Structured feedback is provided to new crew members to help them improve their performance. This feedback also serves as a key input for developing both salespeople and crew members.

[0650] Example 1

[0651] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0652] In conventional role-playing for new crew members, evaluation and feedback of the recorded content was often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of consistency and objectivity in the evaluation, making effective training difficult. Furthermore, the recording data was not stored or analyzed sufficiently, and the data was not fully utilized.

[0653] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0654] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for simultaneously training sales representatives and crew, a means for storing the recorded data in a cloud server, a means for analyzing the stored voice data, and a means for providing the evaluation results from the generative AI model to the user. This enables automatic analysis of the recorded content and provision of feedback, improving the consistency and objectivity of evaluation and realizing effective training. In addition, the recording data is efficiently stored and analyzed, promoting the utilization of data.

[0655] "Means for recording the role-playing content of new crew members" refers to a device or function for acquiring audio of the role-playing performed by new crew members.

[0656] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[0657] "Means of inputting the transcribed content into the generative AI model" refers to the process or function by which the transcribed text data is passed to the generative AI model.

[0658] The "means of systematizing feedback content for the role of sales training professionals" refers to a method or system for organizing and systematizing the feedback provided by sales training experts based on the evaluation results obtained from the generative AI model.

[0659] A "means for simultaneously training sales representatives and crew" is a method or system for simultaneously educating and training both sales representatives and new crew members.

[0660] "Means for storing recorded data on a cloud server" refers to the technology and process for storing acquired voice data on a cloud server via the Internet.

[0661] "Means for analyzing stored voice data" refers to technology or a system for analyzing voice data stored on a cloud server and generating evaluations and feedback.

[0662] "Means for providing evaluation results from a generative AI model to a user" refers to a method or system for communicating the evaluation results generated by the generative AI model to a user.

[0663] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0664] The present invention relates to a system for recording the role-playing of a new crew member, analyzing the content of the recording, and providing feedback. A specific embodiment of this system will be described below.

[0665] System Configuration

[0666] This system is configured using the following hardware and software.

[0667] 1. Hardware:

[0668] Microphone: Used to capture audio for roleplaying of new crew members.

[0669] Smartphone: Use the built-in recording function to capture audio data.

[0670] 2. Software:

[0671] Recording application: Record audio data using a recording application installed on your smartphone.

[0672] Data storage system: The acquired voice data is stored on a cloud server.

[0673] Generative AI model: Analyzes captured audio data and generates role-playing evaluations and feedback.

[0674] System Operation

[0675] Acquiring audio data:

[0676] Before starting role-playing, the user launches the recording application on the smartphone. The device (smartphone) picks up the role-playing audio through the microphone and starts recording. The user performs role-playing and taps the "stop recording" button when finished.

[0677] Audio data storage:

[0678] After the device finishes recording, it uploads the acquired voice data to the cloud server, and the server stores the uploaded voice data in the data storage system.

[0679] Analysis of audio data:

[0680] The server inputs the saved voice data into the generative AI model for analysis. The generative AI model analyzes the voice data and evaluates it on the evaluation items of role-playing (clarity of pronunciation, flow of speech, naturalness of dialogue, and overall evaluation).

[0681] Generate feedback:

[0682] The server receives the evaluation results from the generative AI model and generates feedback to provide to the user. The server sends the generated feedback to the user's device. The user checks the feedback on the device and understands what needs to be improved.

[0683] Examples

[0684] As a concrete example, the following prompt sentence is input into the generative AI model.

[0685] Example prompt:

[0686] "Please analyze the audio data of the new crew member's role-playing and rate it on the following items: 1. Clarity of pronunciation 2. Flow of speech 3. Naturalness of dialogue 4. Overall rating."

[0687] By inputting this prompt sentence into the generative AI model, the model analyzes the voice data and outputs an evaluation for each item. For example, the generative AI model generates feedback like the following:

[0688] Feedback example:

[0689] "Clarity of pronunciation: 8 / 10, flow of speech: 7 / 10, naturalness of dialogue: 9 / 10, overall rating: 8 / 10. Pronunciation was generally clear, but some words were hard to hear. Flow of speech was good, but could have been smoother. Naturalness of dialogue was very high, and overall it was good role-playing."

[0690] In this way, a system is realized in which the server, terminals, and users cooperate to record and evaluate the role-playing of new crew members.

[0691] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0692] Step 1: Prepare to record

[0693] Before starting role-playing, the user launches the recording application on the smartphone. The device displays a pop-up message notifying the completion of the application launch. The input is the user's operation, and the output is the notification that the recording application has been launched.

[0694] Step 2: Acquire audio data

[0695] The user taps the "Start Recording" button on the recording application. The device picks up the role-playing voice in real time through the microphone and starts recording. The user performs the role-playing and taps the "Stop Recording" button when finished. The input is the user's voice, and the output is the recorded voice data.

[0696] Step 3: Save the audio data

[0697] The device temporarily stores the audio data after recording in its internal storage. The device then uploads the stored audio data to a cloud server. The server receives the uploaded audio data and stores it in a data storage system. The input is the recorded audio data, and the output is the audio data stored in the cloud server.

[0698] Step 4: Analyze the audio data

[0699] The server inputs the stored voice data into the generative AI model. The generative AI model analyzes the voice data and evaluates it on the evaluation items of role-playing (clarity of pronunciation, flow of story, naturalness of dialogue, and overall evaluation). The input is the voice data stored on the cloud server, and the output is the evaluation result.

[0700] Step 5: Generate feedback

[0701] The server receives the evaluation results from the generative AI model and generates feedback to provide to the user. The server sends the generated feedback to the user's device. The user checks the feedback on the device and understands future improvements. The input is the evaluation results from the generative AI model, and the output is the feedback provided to the user.

[0702] (Application example 1)

[0703] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0704] It is necessary to provide effective feedback and train new crew members quickly and efficiently in role-playing. However, with conventional methods, analyzing the recorded content and providing feedback is time-consuming and difficult, making it difficult to train efficiently.

[0705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0706] In this invention, the server includes a means for recording the role-playing of newcomers, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for simultaneously training sales representatives and crew members, a means for providing feedback from the generative AI model to new crew members, and a means for pointing out areas for improvement based on the feedback. This allows for quick and effective feedback to be provided to the role-playing of new crew members, enabling efficient training.

[0707] "Means for recording the role-playing content of new crew members" refers to a device or function for acquiring audio of the role-playing performed by new crew members.

[0708] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[0709] "Means for inputting the transcribed content into the generative AI model" refers to means for sending transcribed text data to the generative AI model for analysis and feedback.

[0710] "Means of systematizing feedback content from the perspective of a sales training professional" refers to a means of organizing the feedback obtained from the generative AI model and systematizing it from the perspective of a sales training professional.

[0711] "Means for training sales representatives and crew members simultaneously" refers to means for training both sales representatives and new crew members simultaneously.

[0712] "Means for providing feedback from the generative AI model to new crew members" refers to means for providing feedback to new crew members based on the results of analysis by the generative AI model.

[0713] "Means for pointing out areas for improvement based on feedback" refers to means for specifically pointing out areas for improvement for new crew members based on feedback from the generative AI model.

[0714] The system for implementing the present invention includes a series of processes for recording the role-playing content of a new crew member, transcribing the recorded content, inputting the content into a generative AI model, and providing feedback. A specific embodiment of the system will be described below.

[0715] Hardware and Software Configuration

[0716] The system uses the following hardware and software:

[0717] Hardware: Smartphone microphone

[0718] Software: Python, sounddevice library (audio recording), requests library (HTTP requests)

[0719] Data processing and calculation

[0720] 1. Recording:

[0721] The role-playing of new crew members is recorded in real time using a smartphone microphone.

[0722] The recorded audio data is obtained as a numpy array.

[0723] 2. Transcription:

[0724] The recorded voice data is converted into text information using voice recognition technology.

[0725] The transcribed text data is fed into a generative AI model.

[0726] 3. Analysis using generative AI models:

[0727] The transcribed text data is sent to a generative AI model for analysis.

[0728] The generative AI model analyzes the audio data and generates feedback.

[0729] 4. Providing Feedback:

[0730] The feedback obtained from the generative AI model is provided to new crew members.

[0731] Based on the feedback, we will provide specific areas for improvement for new crew members.

[0732] Examples

[0733] New crew members role-play customer service and record their voices. The recorded data is sent to a generative AI model, which provides feedback such as "your voice tone is too low" or "your greeting to customers is insufficient." This allows new crew members to identify specific areas for improvement and efficiently improve their skills.

[0734] Examples of prompt statements

[0735] Analyze the role-playing audio of the new crew member and provide feedback on the following:

[0736] 1. Tone of voice

[0737] 2. Speaking Speed

[0738] 3. Greeting customers

[0739] 4. Responding to Questions

[0740] In this way, training of new staff members in physical stores can be made more efficient and high quality customer service can be achieved.

[0741] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0742] Step 1:

[0743] The terminal (smartphone) records the role-playing content of the new crew member. The recording is done using the smartphone's microphone, and the audio data is obtained as a numpy array. The input is the new crew member's voice, and the output is the recorded audio data.

[0744] Step 2:

[0745] The device converts the recorded voice data into text information using voice recognition technology. Specifically, the voice recognition software analyzes the voice data and generates corresponding text data. The input is the voice data, and the output is the transcribed text data.

[0746] Step 3:

[0747] The terminal sends the transcribed text data to the generative AI model. Specifically, the terminal sends the text data as an HTTP request to the API of the generative AI model. The input is the transcribed text data, and the output is the request sent to the generative AI model.

[0748] Step 4:

[0749] The server (generative AI model) analyzes the received text data and generates feedback. The generative AI model evaluates the text data based on the prompt sentence and generates specific feedback. The input is the text data and the output is the feedback content.

[0750] Step 5:

[0751] The server sends the generated feedback back to the terminal. Specifically, it sends the feedback content to the terminal as an HTTP response. The input is the feedback content from the generative AI model, and the output is the feedback sent to the terminal.

[0752] Step 6:

[0753] The terminal provides the received feedback to the new crew member. Specifically, the feedback content is displayed on a display screen so that the new crew member can check it. The input is the feedback content from the server, and the output is the feedback provided to the new crew member.

[0754] Step 7:

[0755] The user (new crew member) identifies areas for improvement based on the provided feedback and reflects them in the next role-playing. Specifically, the user checks the feedback and performs the role-playing while keeping in mind the areas for improvement pointed out. The input is the feedback, and the output is the improved role-playing.

[0756] Example 2

[0757] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0758] In conventional systems that convert voice data into text information, the process from acquiring voice data to converting it into text information and then using it was complicated, making it difficult to carry out efficiently. There were also issues with the accuracy of converting voice data into text information using voice recognition technology, and with the management and use of the converted data. This caused problems with the time and effort required when users tried to use it as a prompt to input into a generative AI model.

[0759] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice data, a means for transmitting the acquired voice data to the server, a means for converting the voice data into character information, a means for storing the converted character information in a database, and a means for inputting the stored character information into a generative AI model. This makes it possible to efficiently perform the process from acquiring voice data to converting it into character information and using it.

[0760] "Audio data" refers to data in which audio is recorded in digital format.

[0761] "Means of acquisition" refers to devices and software for collecting voice data.

[0762] "Transmitting means" refers to a communication means for transferring the acquired voice data to a server.

[0763] "Means for converting into text information" refers to technology that analyzes voice data and generates corresponding text data.

[0764] "Means of storing in a database" refers to a system for systematically storing the converted character information.

[0765] "Means of input to the generative AI model" refers to the interfaces or processes used to provide the stored textual information to the AI ​​model.

[0766] "Means of recording in real time" refers to devices or software for instantly recording audio as digital data.

[0767] "Speech recognition technology" refers to technology for analyzing voice data and generating corresponding text information.

[0768] The present invention is a system for efficiently converting voice data into text information and inputting the text information into a generative AI model. A specific embodiment of the system will be described below.

[0769] First, a user uses a terminal for acquiring voice data. The terminal includes a voice input device such as a smartphone and a microphone. The user uses these devices to record the voice data. For example, when recording the contents of a meeting, the user can use a recording app on the smartphone.

[0770] The device then transmits the captured voice data to a server over the Internet, using HTTP requests to upload the voice data to the server, which then runs software to process the received voice data.

[0771] The server converts the received voice data into text information. This conversion is performed using voice recognition technology. Specifically, a voice recognition API or voice recognition service can be used. For example, Google's voice recognition API or IBM's Watson can be used. The server sends the voice data to these services and receives the text information in return.

[0772] The server then stores the converted text in a database, along with other metadata such as user ID and timestamp, to make it easier for users to search for the data later.

[0773] Finally, the user uses the saved text information as a prompt to be input to the generative AI model. The user opens a web browser and logs into the system's web interface. The user accesses the "Meeting Records" page and checks the saved text information. If necessary, the user copies the text information and uses it as a prompt to be input to the generative AI model.

[0774] As a concrete example, consider a case where a user wants to record the audio of a meeting and save the content as text information. The user uses a smartphone to record the audio of the meeting and sends the audio data to a server. The server uses a speech recognition API to convert the audio data into text information and stores the text information in a database. Later, the user can use the text information as a prompt sentence to input into a generative AI model.

[0775] An example of a prompt might be:

[0776] "Please summarize the following meeting: 'In today's meeting, we discussed the launch date of the new product. The marketing team proposed to set the launch date on the 15th of next month, but the development team said that it is not ready yet.'"

[0777] In this way, users can efficiently convert voice data into textual information that can be used in generative AI models.

[0778] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0779] Step 1:

[0780] A user obtains audio data using a smartphone or microphone. Specifically, the user opens a recording app on the smartphone and taps the "Start Recording" button to record the audio of the meeting. When recording is finished, the user taps the "Stop Recording" button to save the audio data. The input is audio, and the output is digital audio data.

[0781] Step 2:

[0782] The terminal transmits the acquired voice data to the server. Specifically, the user selects the saved voice data from the recording app and taps the "Send" button. This uploads the voice data to the server via an HTTP request. The input is the digital voice data, and the output is the voice data transmitted to the server.

[0783] Step 3:

[0784] The server converts the received voice data into text information. Specifically, the server sends the voice data to a voice recognition API, which analyzes the voice data and generates corresponding text information. For example, the server sends voice data to a voice recognition service and receives the returned text information. The input is voice data and the output is text information.

[0785] Step 4:

[0786] The server saves the converted text information in a database. Specifically, the server inserts the received text information into the "meeting records" table in the database. At this time, metadata such as the user ID and timestamp are also saved. The input is text information and metadata, and the output is the text information saved in the database.

[0787] Step 5:

[0788] The user uses the saved text information as a prompt sentence to be input to the generative AI model. Specifically, the user opens a web browser and logs into the system's web interface. The user accesses the "Meeting Records" page and checks the saved text information. If necessary, the user copies this text information and uses it as a prompt sentence to be input to the generative AI model. The input is the text information obtained from the database, and the output is the prompt sentence to be input to the generative AI model.

[0789] (Application example 2)

[0790] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0791] In conventional food delivery services, when delivery personnel manually input order details from customers, input errors and time loss were an issue. In addition, delivery personnel had to use their hands to input the order details, which made it difficult to carry out work efficiently.

[0792] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for acquiring voice data, a means for converting the acquired voice data into text information, and a means for registering the converted text information in the order system. This makes it possible to efficiently process the order details input by voice by the delivery person.

[0793] "Audio data" refers to data in which audio is recorded in digital format.

[0794] "Means for acquiring" refers to a device or method for collecting audio data.

[0795] "Text information" refers to information obtained by converting voice data into text.

[0796] "Means of converting" refers to technology or equipment for converting voice data into text information.

[0797] "Order System" refers to a system for managing and processing orders from customers.

[0798] "Means for registering" refers to a method or device for inputting and storing converted character information into the ordering system.

[0799] "Management Means" means the methods and devices used to organize, track and manage orders.

[0800] A "delivery person" refers to a person whose role is to deliver goods to customers.

[0801] "Mobile device" refers to an electronic device that can be carried by a delivery person.

[0802] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text information.

[0803] To implement this invention, you need a mobile terminal for acquiring voice data, voice recognition technology for converting the voice data into text information, and a means for registering the converted text information in the ordering system.

[0804] System Configuration

[0805] 1. Mobile terminal: A mobile terminal such as a smartphone or smart glasses is used by the delivery person. This allows the delivery person to input the order details from the customer by voice.

[0806] 2. Speech recognition technology: To convert voice data into text information, we use voice recognition services such as Google's speech recognition API and IBM's Watson, which allows us to convert voice data into text information with high accuracy.

[0807] 3. Ordering system: An ordering system is used to manage and process the converted text information. This system has the function of automatically registering and managing the order contents.

[0808] Program Processing

[0809] The server first receives voice data from the mobile device. It then converts the voice data into text using voice recognition technology. The converted text is automatically registered in the order system. This allows delivery personnel to input order details without using their hands, making work more efficient.

[0810] Examples

[0811] For example, if a delivery person voice-inputs "one pizza, two colas," the mobile device captures this voice data. Next, the voice data is converted into text information using voice recognition technology, and registered in the ordering system as "one pizza, two colas." This series of processes allows the delivery person to quickly and accurately register the order details in the system.

[0812] Examples of prompt statements

[0813] "Please convert the following audio data into text: 1 pizza, 2 cokes"

[0814] In this way, a series of processes can be realized, from converting voice data into text information and registering it in the order system. This system improves the work efficiency of delivery personnel and reduces the risk of input errors.

[0815] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0816] Step 1:

[0817] The user (delivery person) inputs voice data using a mobile terminal. Specifically, the delivery person speaks the order details into a smartphone or smart glasses. The input is recorded as voice data on the mobile terminal.

[0818] Step 2:

[0819] The terminal acquires the voice data. The acquired voice data is temporarily stored in the memory of the terminal. The input is the delivery person's voice data, and the output is the voice data stored in the terminal.

[0820] Step 3:

[0821] The terminal transmits the acquired voice data to the server. Specifically, the terminal uploads the voice data to the server via the Internet. The input is the voice data stored in the terminal, and the output is the voice data transmitted to the server.

[0822] Step 4:

[0823] The server receives the voice data and converts it into text information using voice recognition technology. Specifically, the server calls Google's voice recognition API or IBM's Watson to convert the voice data into text information. The input is the voice data sent to the server, and the output is the converted text information.

[0824] Step 5:

[0825] The server registers the converted text information in the ordering system. Specifically, the server saves the text information in the ordering system's database and registers it as the order details. The input is the converted text information, and the output is the order details registered in the ordering system.

[0826] Step 6:

[0827] The ordering system manages the registered order details. Specifically, the ordering system organizes the order details and sends notifications to delivery personnel and customers as necessary. The input is the order details registered in the ordering system, and the output is the managed order information.

[0828] Step 7:

[0829] The user (delivery person) receives a notification from the ordering system and confirms the order details. Specifically, the delivery person receives a notification from the ordering system through a mobile terminal and confirms the order details. The input is the notification from the ordering system, and the output is the order details confirmed by the delivery person.

[0830] Example 3

[0831] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0832] In conventional sales training systems, it was difficult to effectively provide feedback and systematize the role-playing content of new recruits. In addition, the quality of the generated conversations and the accuracy of the feedback were low, making it inefficient to simultaneously train sales representatives and crew members. This limited the effectiveness of the training and made it difficult to improve practical skills.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0834] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for receiving the generated conversation, a means for systematizing the feedback content of the generated conversation using text analysis and natural language processing technology, and a means for simultaneously training the sales representative and the crew. This improves the quality of the generated conversation, increases the accuracy of the feedback, and makes it possible to efficiently train the sales representative and the crew at the same time.

[0835] "Recording means" refers to a device or software that records the role-playing content of a newcomer as audio data.

[0836] A "transcription means" is a technology or device that converts recorded audio data into text data.

[0837] A "generative AI model" is an artificial intelligence algorithm or system that generates natural-sounding conversations based on input text information.

[0838] A "means for receiving the generated conversation" is a device or software that obtains the conversation data output from the generative AI model.

[0839] "Text analysis" is a technique or method for analyzing text data and understanding its meaning and structure.

[0840] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[0841] A "means for systematizing feedback content" is a technique or device for structuring and organizing feedback based on the generated conversation as a professional role in sales training.

[0842] A "means for simultaneously training salespeople and crew" is a system or method for effectively training salespeople and new crew members simultaneously using generated conversations and feedback.

[0843] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0844] This invention is a system that records the role-playing content of new recruits, transcribes the recording content, inputs it into a generative AI model, and organizes the generated conversation as feedback. A specific embodiment of this system will be described below.

[0845] Server Roles

[0846] The server receives text information entered by the user. Specifically, the server receives an HTTP request and extracts text information from the request body. Next, the server inputs the received text information into a generative AI model. For example, OpenAI's GPT-4 can be used as this generative AI model. The server sends the text information to the generative AI model as an API request and receives the generated conversation. The generated conversation is analyzed using text analysis and natural language processing technology, and the feedback content is systematized. For this analysis, Python's NLTK library or SpaCy can be used.

[0847] Terminal Roles

[0848] The terminal transmits the character information input by the user to the server. Specifically, the terminal transmits the character information input by the user to the server as an HTTP request. After receiving the generated conversation from the server, the terminal displays the conversation to the user. Furthermore, the terminal provides an interface for the user to input feedback. This allows the user to input feedback for the generated conversation.

[0849] User Roles

[0850] The user inputs text information using the terminal. Specifically, the user inputs text information using a keyboard or a touch screen of the terminal. The user checks the generated conversation on the terminal and inputs feedback as necessary. The user inputs feedback using a feedback input interface of the terminal and checks that the content is organized.

[0851] Examples

[0852] For example, in a sales training scenario, a user (sales trainee) types into the terminal, "Please explain the features of the new product." The terminal sends this text information to the server, which then sends the text information to GPT-4 as an API request. GPT-4 generates a dialogue such as, "The features of the new product are that it is lightweight, easy to carry, and has a long battery life." The server receives the generated dialogue and systematizes the feedback content using text analysis and natural language processing techniques. The terminal displays the generated dialogue and feedback content to the user, who then checks it and enters feedback.

[0853] Examples of prompt statements

[0854] Prompt: "Describe the features of your new product."

[0855] When this prompt sentence is input to GPT-4, the AI ​​generates a natural conversation and provides useful feedback for the sales training scenario. The flow of the specific process in Example 3 is explained with reference to FIG.

[0856] Program processing flow

[0857] Step 1: Enter text information

[0858] Subject: User

[0859] The user uses the terminal to input text information. Specifically, the user uses the keyboard or touch screen of the terminal to input text information such as "Please explain the features of the new product." This input text information becomes the input data for the next step.

[0860] Step 2: Send text information

[0861] Subject: Terminal

[0862] The terminal sends the character information entered by the user to the server. Specifically, the terminal generates an HTTP request, includes the character information in the request body, and sends it to the server. The input of this step is the character information entered by the user, and the output is the HTTP request sent to the server.

[0863] Step 3: Receiving text information

[0864] Subject: Server

[0865] The server receives the HTTP request sent from the terminal and extracts text information from the request body. Specifically, the server analyzes the HTTP request and obtains the text information. The input of this step is the HTTP request from the terminal, and the output is the extracted text information.

[0866] Step 4: Inputting text information into the AI ​​model

[0867] Subject: Server

[0868] The server inputs the extracted character information into the generative AI model. Specifically, the server generates an API request to the generative AI model (e.g., GPT-4) and sends the character information. The input of this step is the extracted character information, and the output is an API request to the generative AI model.

[0869] Step 5: Receive the generated conversation

[0870] Subject: Server

[0871] The server receives the generated conversation from the generative AI model. Specifically, the server receives the API response from the generative AI model and extracts the generated conversation. The input of this step is the API response from the generative AI model, and the output is the generated conversation.

[0872] Step 6: Organize your feedback

[0873] Subject: Server

[0874] The server organizes the feedback content of the generated conversation using text analysis and natural language processing techniques. Specifically, the server analyzes the generated conversation and organizes the feedback content using Python's NLTK library and SpaCy. The input of this step is the generated conversation, and the output is the organized feedback content.

[0875] Step 7: Receive the generated conversation

[0876] Subject: Terminal

[0877] The terminal receives the generated conversation from the server. Specifically, the terminal receives the HTTP response from the server and extracts the generated conversation. The input of this step is the HTTP response from the server, and the output is the generated conversation.

[0878] Step 8: View the generated conversation

[0879] Subject: Terminal

[0880] The terminal displays the generated conversation to the user. Specifically, the terminal displays the generated conversation on a user interface. The input of this step is the generated conversation, and the output is the conversation displayed to the user.

[0881] Step 9: Provide feedback

[0882] Subject: User

[0883] The user inputs feedback for the generated conversation. Specifically, the user inputs feedback through a feedback input interface of the terminal. The input of this step is the generated conversation, and the output is the feedback input by the user.

[0884] Step 10: Review the feedback

[0885] Subject: User

[0886] The user confirms that the feedback content is organized. Specifically, the user confirms the result of the feedback content organization displayed on the terminal screen. The input of this step is the organized feedback content, and the output is the confirmed feedback content.

[0887] (Application example 3)

[0888] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0889] Conventional sales training systems lacked a means to effectively provide feedback on the role-playing content of new employees, which led to slow improvement in sales skills. In addition, because the feedback content was not systematized, it was difficult to provide appropriate guidance to individual sales representatives. Furthermore, there was a need for an efficient training method that utilizes smartphones.

[0890] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0891] In this invention, the server includes a means for recording the role-playing content of new recruits, a means for transcribing the recorded content, a means for inputting the transcribed content into a generative AI, a means for systematizing the feedback content as a professional role of sales training, a means for simultaneously training sales representatives and crew members, a means for analyzing the generated conversation using text analysis and natural language processing technology and providing feedback, and a means including an application to be installed on a smartphone. This makes it possible to improve the efficiency of sales training and provide effective feedback.

[0892] "Means for recording the role-playing content of new recruits" refers to devices or software for recording the audio of the role-playing performed by new recruits.

[0893] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[0894] "Means of inputting the transcribed content into the generative AI" refers to the interface or process for inputting the transcribed text data into the chat generative AI.

[0895] "Means of systematizing feedback content as a sales training professional" refers to techniques and methods for analyzing the generated conversation content from the perspective of a sales training professional and structuring the feedback.

[0896] A "means of simultaneous salesperson and crew development" is a system or method by which salespeople and new crew members learn and improve their skills at the same time.

[0897] "Means for analyzing the generated conversation using text analysis and natural language processing technology and providing feedback" refers to technologies and methods for evaluating the content of the generated conversation using text analysis and natural language processing technology and providing feedback based on the results.

[0898] "Means including an application installed on a smartphone" refers to a system or software including an application that runs on a smartphone.

[0899] A system for implementing this invention has the following configuration. First, a smartphone or a dedicated recording device is used as a means for recording the role-playing content of the new recruit. The recorded voice data is converted into text information using voice recognition technology. Google Speech-to-Text API, IBM Watson Speech to Text, etc. can be used for this voice recognition technology.

[0900] Next, the transcribed text data is input into a chat generation AI. For this chat generation AI, OpenAI's GPT-3 is used. GPT-3 can generate natural conversations based on the input text data.

[0901] The generated conversation content is analyzed using text analysis and natural language processing technology. Specifically, libraries such as TextBlob and NLTK (Natural Language Toolkit) are used to evaluate the positivity and subjectivity of the conversation. Based on the results of this analysis, feedback is provided from the perspective of sales training professionals.

[0902] This feedback is provided through an application installed on a smartphone. The application provides an interface for salespeople and new crew members to simultaneously learn and improve their skills. For example, it displays the conversations generated and their feedback, and allows users to evaluate their own performance.

[0903] As a concrete example, if a salesperson were simulating a conversation with a customer looking for a new smartphone, they might use the following prompt:

[0904] "Customer: Hello, I'm looking for a new smartphone today. Which model would you recommend?"

[0905] If we feed this prompt into a generative AI model, it might generate the following dialogue:

[0906] “Customer: Hello, I’m looking for a new smartphone today. Which model would you recommend?

[0907] Staff: Hello, customer. I understand you are looking for a new smartphone. We can recommend some models based on your needs. For example, the latest model is very popular and has a great camera. We can also recommend other high-performance models. What features do you care about?

[0908] The conversation is analyzed by TextBlob and feedback is provided. For example, feedback such as "Positiveness: 0.5, Subjectivity: 0.3" can be obtained. Based on this feedback, sales staff can improve their responses.

[0909] As described above, the present invention realizes efficient sales training and effective feedback provision by combining the means of recording, transcription, chat generation, text analysis, and feedback provision.

[0910] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0911] Step 1:

[0912] A user starts role-playing using a smartphone.

[0913] Input: User's voice

[0914] Output: Recorded audio data

[0915] Specific operation: The user activates the recording function of the smartphone and starts role-playing. The smartphone records the user's voice.

[0916] Step 2:

[0917] The server receives the recorded voice data and converts it into text information using voice recognition technology.

[0918] Input: Recorded audio data

[0919] Output: Transcribed text data

[0920] Specific operation: The server uses the Google Speech-to-Text API or IBM Watson Speech to Text to convert the voice data into text data.

[0921] Step 3:

[0922] The server inputs the transcribed text data into a chat generation AI, which generates natural conversation.

[0923] Input: Transcribed text data

[0924] Output: Generated conversation text

[0925] How it works: The server uses OpenAI's GPT-3 to generate natural-sounding conversations based on input text data.

[0926] Step 4:

[0927] The server analyzes the generated conversation text using text analysis and natural language processing techniques and provides feedback.

[0928] Input: Generated conversation text

[0929] Output: Feedback data (positivity, subjectivity, etc.)

[0930] Specific operation: The server uses TextBlob and NLTK to analyze the generated conversation text and evaluate its positivity and subjectivity.

[0931] Step 5:

[0932] The server sends the feedback data to the smartphone and displays it to the user.

[0933] Input: Feedback data

[0934] Output: Feedback information displayed to the user

[0935] Specific operation: The server sends the feedback data to the smartphone, and the smartphone application displays the feedback information to the user.

[0936] Step 6:

[0937] Users can view feedback and use it to improve their sales skills.

[0938] Input: Feedback information displayed to the user

[0939] Output: Understanding users and improving their behavior

[0940] Specific Actions: Users review the feedback information displayed on their smartphone screen, evaluate their sales skills, and identify areas for improvement.

[0941] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0942] "Example 1"

[0943] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system records the role-playing of a new crew member and transcribes the recorded content. The transcribed content is input to a generative AI, which organizes the feedback content as a professional role in sales training. Furthermore, this system includes an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from the recorded content and reflects the results in the feedback content. Specifically, the emotion is estimated from the user's tone of voice, speaking style, choice of words, etc., and the information is used in the feedback. This enables more specific and useful feedback that also takes into account the emotional reactions of the new crew member.

[0944] "Example 2"

[0945] The emotion engine also optimizes the training of salespeople and crew members based on the results of emotion analysis. Specifically, it analyzes how new crew members emotionally react to certain scenarios and how those reactions affect sales performance, and adjusts the training program accordingly. For example, if it senses that new crew members are anxious about tough questions from customers, it can provide training to strengthen their ability to handle such scenarios.

[0946] "Example 3"

[0947] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system records the role-playing of a new crew member and transcribes the recorded content. The transcribed content is input to a generative AI, which organizes the feedback content as a professional role in sales training. Furthermore, this system includes an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from the recorded content and reflects the results in the feedback content. Specifically, the emotion is estimated from the user's tone of voice, speaking style, choice of words, etc., and the information is used in the feedback. This enables more specific and useful feedback that also takes into account the emotional reactions of the new crew member.

[0948] The process flow of each embodiment will be described below.

[0949] "Example 1"

[0950] Step 1: Have new crew members role-play and record the session.

[0951] Step 2: Transcribe the recorded session.

[0952] Step 3: Input the transcribed content into the generative AI.

[0953] Step 4: The generative AI generates the feedback content.

[0954] Step 5: The emotion engine analyzes the user's emotions from the recording.

[0955] Step 6: The emotion engine applies the analysis results to the feedback content.

[0956] "Example 2"

[0957] Step 1: The emotion engine analyzes the user's emotions from the recording.

[0958] Step 2: The sentiment engine uses the results of sentiment analysis to optimize the training of sales representatives and crew.

[0959] Step 3: The emotion engine analyzes how the new crew member will emotionally react to a particular scenario.

[0960] Step 4: The emotion engine analyzes how that response impacts sales outcomes.

[0961] Step 5: The emotion engine adjusts the training program based on the analysis results.

[0962] Example 1

[0963] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0964] In the past, role-playing for new crew members required the manual process of transcribing recordings and providing feedback, which was time-consuming and labor-intensive. In addition, the feedback did not take into account emotional reactions, limiting the improvement of response skills in actual work. Furthermore, there was a lack of effective means to simultaneously train salespeople and crew members.

[0965] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the newcomer, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for performing emotion analysis using an emotion engine that recognizes the user's emotions, a means for reflecting the emotion analysis result in the feedback content, and a means for simultaneously training the salesperson and the crew. This makes it possible to automate the process from recording to providing feedback and provide specific and useful feedback that takes into account emotional reactions. In addition, it is possible to effectively train the salesperson and the crew at the same time.

[0966] "Means for recording the role-playing content of new crew members" refers to devices or functions for recording the audio of the role-playing performed by new crew members.

[0967] A "means of transcribing the recorded content" refers to technology or software for converting recorded audio data into text data.

[0968] "Means for inputting the transcribed content into the generative AI model" refers to the process of sending the transcribed text data to the generative AI model to generate feedback content.

[0969] "A means of systematizing feedback content for the role of a sales training professional" refers to a method that uses a generative AI model to structure and provide feedback like that given by a sales training professional.

[0970] The "means for performing emotion analysis using an emotion engine that recognizes the user's emotions" refers to software or algorithms for analyzing emotions from a user's voice or text.

[0971] The "means for reflecting the emotion analysis result in the feedback content" is a method for incorporating the emotion analysis result obtained by the emotion engine into the generated feedback content.

[0972] A "means of simultaneously developing salespeople and crew" is an educational program or system that allows salespeople and new crew members to learn and grow at the same time.

[0973] This invention is a system that effectively records the role-playing of new crew members, transcribes the contents, and provides feedback. Furthermore, by analyzing the user's emotions and reflecting the results in the feedback, it is possible to provide more specific and useful feedback.

[0974] Hardware and Software Used

[0975] Recording device

[0976] When the user role-plays a new crew member, the user can obtain voice data using a microphone or the recording function of the smartphone. For example, the user can use a voice memo app on the smartphone or a dedicated recording device.

[0977] Speech Recognition Software

[0978] The server uses speech recognition software to transcribe the captured audio data, specifically using a speech recognition service such as the Google Cloud Speech-to-Text API. This process converts the audio data into text data.

[0979] Generative AI Models

[0980] The server inputs the transcribed text data into a generative AI model, such as OpenAI's GPT-3, which then organizes the feedback into a professional role in sales training.

[0981] Sentiment Analysis Tools

[0982] The server uses an emotion engine to analyze the user's emotions from the recordings, using emotion analysis tools such as IBM Watson Tone Analyzer, which infers emotions from the user's tone of voice, speaking style, and choice of words.

[0983] Examples

[0984] New crew members role-play customer service and record the content on their smartphones. The recorded voice data is sent to a server and transcribed using Google Cloud Speech-to-Text API. The transcribed text data is then input into GPT-3, which generates feedback content as a professional role in sales training. In addition, sentiment analysis is performed using IBM Watson Tone Analyzer, and the results are reflected in the feedback.

[0985] Examples of prompt statements

[0986] "A new crew member has role-played interacting with a customer. Please transcribe the audio data below and generate feedback. Also, please perform sentiment analysis and incorporate the results into your feedback."

[0987] In this way, users can get specific, useful feedback on their role-playing of new crew members that also takes into account their emotional reactions.

[0988] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0989] Step 1:

[0990] When a user role-plays a new crew member, the user acquires voice data using a microphone or the recording function of the smartphone. Specifically, the user opens a voice memo app on the smartphone and presses the "Start Recording" button to record the role-playing. When the recording is finished, the user presses the "Stop Recording" button to save the voice data. The input is the voice of the role-playing, and the output is the recorded voice data.

[0991] Step 2:

[0992] The terminal uploads the recorded voice data to the server. The user performs this operation by selecting the voice file through a dedicated application or a web interface and pressing the "upload" button. The input is the recorded voice data, and the output is the voice data uploaded to the server.

[0993] Step 3:

[0994] The server receives the uploaded voice data and transcribes it using a speech recognition service such as Google Cloud Speech-to-Text API. The server sends the voice data to the API and stores the returned text data. The input is the uploaded voice data, and the output is the transcribed text data.

[0995] Step 4:

[0996] The server inputs the transcribed text data into a generative AI model (e.g., OpenAI's GPT-3). The server sends the text data to the generative AI model as a prompt sentence to generate feedback content. The input is the transcribed text data, and the output is the generated feedback content.

[0997] Step 5:

[0998] The server uses a sentiment analysis tool such as IBM Watson Tone Analyzer to analyze user sentiment from the transcribed text data. The server sends the text data to the sentiment analysis tool and stores the returned sentiment analysis results. The input is the transcribed text data, and the output is the sentiment analysis results.

[0999] Step 6:

[1000] The server integrates the feedback content obtained from the generative AI model with the sentiment analysis results. The server reflects the sentiment analysis results in the feedback content to generate the final feedback. The input is the generated feedback content and the sentiment analysis results, and the output is the integrated final feedback.

[1001] Step 7:

[1002] The server provides the final feedback to the user, who can review the feedback through a dedicated application or a web interface. The input is the integrated final feedback, and the output is the feedback provided to the user.

[1003] (Application example 1)

[1004] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1005] In the conventional training system for new crew members, the content of role-playing was recorded and transcribed, making it difficult to analyze emotions and improve the quality of feedback. In addition, the feedback was one-sided and it was not possible to provide specific advice that took emotional reactions into account. This made it difficult to effectively support the growth of new crew members.

[1006] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the new employee, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for analyzing the user's emotions using an emotion engine, a means for reflecting the results of the emotion analysis in the feedback content, and a means for simultaneously training the sales representative and the crew. This makes it possible to provide specific and useful feedback that takes into account emotional reactions.

[1007] A "newcomer" refers to an employee who has been newly hired by a company or organization.

[1008] "Role playing" refers to a training method in which actual tasks or situations are simulated by playing specific roles.

[1009] "Recording" refers to the act of saving sound on a recording medium.

[1010] "Transcription" refers to the process of converting audio data into text data.

[1011] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or data.

[1012] "Feedback" refers to the act of providing evaluation and suggestions for improvement for specific actions or results.

[1013] An "emotion engine" refers to a system that analyzes emotions from voice and text and outputs the results.

[1014] "Sales training" refers to the training and education provided to sales representatives to acquire the necessary knowledge and skills.

[1015] A "professional role" refers to a role played by someone with specialized knowledge or experience.

[1016] "Systematization" refers to organizing and structuring information and data.

[1017] "Development" refers to education and training to improve specific skills and abilities.

[1018] A system for carrying out this invention has the following configuration. First, a user performs role-playing using a smartphone. The smartphone's microphone is used to record the voice of the role-playing. The recorded voice is saved using a recording application in the smartphone (e.g., Android (registered trademark)'s MediaRecorder class).

[1019] Next, the recorded voice file is transcribed using a speech recognition API (e.g., Google Cloud Speech-to-Text API). The transcribed text is input to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions. The emotion engine estimates emotions from the tone of voice, speaking style, and choice of words, and outputs the results.

[1020] The transcribed text and the results of the sentiment analysis are then input into a generative AI model (e.g., OpenAI GPT-3), which uses this information to generate feedback that is then displayed in the smartphone application UI and provided to the user.

[1021] As a concrete example, by inputting the following prompt sentence into the generative AI model, feedback reflecting the results of sentiment analysis can be obtained.

[1022] Prompt statement:

[1023] Below is the text of a role-playing session for a new crew member. Please provide your feedback on this, taking into account the results of the sentiment analysis.

[1024] Role-playing content:

[1025] "Hello, welcome. What item would you like to purchase today?"

[1026] Sentiment analysis results:

[1027] Tone: Friendly

[1028] Emotion: Joy

[1029] feedback:

[1030] By inputting this prompt sentence into the generative AI model, feedback such as "Your greeting was very friendly and left a good impression on the customer. Next time, it would be even better if you added a more specific question." In this way, new crew members can effectively train in physical stores.

[1031] The system makes it possible to provide specific, useful feedback that takes emotional responses into account, effectively supporting the development of new crew members.

[1032] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1033] Step 1:

[1034] The user uses a smartphone to perform role-playing. The smartphone's microphone is used to record the audio of the role-playing. A recording application (e.g. Android (registered trademark) MediaRecorder class) is launched, and recording begins when the user presses the record button. The recorded audio is saved in the smartphone as an audio file.

[1035] Input: User's voice

[1036] Output: Recorded audio file

[1037] Step 2:

[1038] The device sends the recorded audio file to a speech recognition API (e.g., Google Cloud Speech-to-Text API). The speech recognition API converts the audio data into text data and returns the transcribed text. The device receives this text data.

[1039] Input: Recorded audio file

[1040] Output: Transcribed text data

[1041] Step 3:

[1042] The terminal inputs the transcribed text data into an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the user's emotion from the text data and returns the emotion analysis result. The terminal receives this emotion analysis result.

[1043] Input: Transcribed text data

[1044] Output: Sentiment analysis results

[1045] Step 4:

[1046] The device inputs the transcribed text data and the sentiment analysis results into a generative AI model (e.g., OpenAI GPT-3). The generative AI model generates feedback based on this information. The device receives this feedback.

[1047] Input: Transcribed text data, sentiment analysis results

[1048] Output: The generated feedback

[1049] Step 5:

[1050] The terminal displays the generated feedback on the application UI, and the user can check the feedback through the smartphone screen.

[1051] Input: Generated feedback

[1052] Output: Feedback displayed in the application UI

[1053] In this way, users can receive specific, useful feedback that takes into account their emotional reactions to the role-playing content.

[1054] Example 2

[1055] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1056] With traditional training methods for new crew members, it was difficult to provide optimal training based on individual emotional responses and sales results. In addition, the process of converting voice data into text information and analyzing emotions was often done manually and was inefficient. This limited the effectiveness of the training program and prevented crew members from maximizing their growth.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1058] In this invention, the server includes a means for acquiring voice data, a means for transmitting the acquired voice data to the server, a means for converting the voice data into text information, a means for sentiment analysis of the text information, and a means for optimizing the training program based on the result of the sentiment analysis. This makes it possible to efficiently perform a series of steps from automatic acquisition of voice data, conversion to text information, sentiment analysis, and optimization of the training program.

[1059] "Audio data" refers to data in which audio is recorded in digital format.

[1060] A "server" is a computer system that processes and stores data over a network.

[1061] "Text information" refers to voice data converted into text format.

[1062] "Sentiment analysis" is the process of analyzing text data for emotions and emotional trends.

[1063] A "development program" is a training plan designed to improve the skills and knowledge of new crew members.

[1064] "Means for acquiring" refers to a device or method for collecting voice data.

[1065] The "transmitting means" refers to a device or method for transferring the acquired voice data to the server.

[1066] The "converting means" refers to a device or method for converting voice data into text information.

[1067] An "optimizing means" is a device or method for adjusting a training program based on the results of sentiment analysis.

[1068] The present invention is a system for acquiring voice data, converting it into text information, performing sentiment analysis, and optimizing a training program based on the results. A specific embodiment of this system will be described below.

[1069] First, the user uses a device such as a smartphone or tablet to record a conversation with a customer. The user starts the recording app on the device and presses the "Start Recording" button to record the conversation. When the conversation is over, the user presses the "Stop Recording" button to end the recording.

[1070] Next, the device transmits the recorded voice data to the server. The device selects the recording file and presses the "send" button to upload the voice data to the server via the Internet.

[1071] The server converts the received voice data into text using a voice recognition service such as Google's voice recognition API or IBM's Watson. The server sends the voice data to the voice recognition API, which analyzes the voice data and generates text information. The generated text information is then sent back to the server.

[1072] The server then inputs the text information into the emotion engine to perform emotion analysis. The server sends the text information to the emotion engine, which analyzes the text information and generates emotion analysis results. The results are sent back to the server.

[1073] Finally, the server optimizes the training program for new crew members based on the results of the sentiment analysis. The server evaluates the results of the sentiment analysis and analyzes how new crew members emotionally react to specific scenarios. For example, if it is analyzed that a new crew member feels anxious when asked a tough question by a customer, the server generates a training program to strengthen the ability to respond to that scenario.

[1074] As a concrete example, a user uses a smartphone to record a conversation with a customer, converts the voice data into text, and then performs sentiment analysis on the text. The following is an example of a prompt sentence that optimizes the training program for new crew members based on the results.

[1075] Example prompt:

[1076] "Record conversations with customers and convert the audio data into text. Then run the text through sentiment analysis and use the results to optimize your onboarding program."

[1077] In this way, the system can efficiently perform a series of processes from acquiring voice data to analyzing emotions and optimizing training programs.

[1078] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1079] Step 1:

[1080] Acquiring audio data

[1081] The user uses a device such as a smartphone or tablet to record a conversation with a customer. The user starts the recording app on the device and presses the "Start Recording" button to record the conversation. When the conversation is over, the user presses the "Stop Recording" button to end the recording.

[1082] Input: Customer conversation

[1083] Output: Recorded audio data

[1084] Step 2:

[1085] Sending voice data to the server

[1086] The device sends the recorded voice data to the server. The device selects the recording file and presses the "Send" button to upload the voice data to the server via the Internet.

[1087] Input: Recorded audio data

[1088] Output: Audio data uploaded to the server

[1089] Step 3:

[1090] Converting voice data into text

[1091] The server converts the received voice data into text using a voice recognition service such as Google's voice recognition API or IBM's Watson. The server sends the voice data to the voice recognition API, which analyzes the voice data and generates text information. The generated text information is then sent back to the server.

[1092] Input: Audio data uploaded to the server

[1093] Output: Text information

[1094] Step 4:

[1095] Conducting sentiment analysis

[1096] The server inputs the text information into the emotion engine and performs sentiment analysis. The server sends the text information to the emotion engine, which analyzes the text information and generates the result of the sentiment analysis. The result is sent back to the server.

[1097] Input: Text information

[1098] Output: Sentiment analysis results

[1099] Step 5:

[1100] Optimizing training programs

[1101] The server optimizes the training program for new crew members based on the results of the sentiment analysis. The server evaluates the results of the sentiment analysis and analyzes how new crew members react emotionally to specific scenarios. For example, if it is analyzed that a new crew member feels anxious when asked a tough question by a customer, a training program is generated to strengthen the ability to respond to that scenario.

[1102] Input: Sentiment analysis results

[1103] Output: Optimized breeding program

[1104] (Application example 2)

[1105] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1106] In conventional sales training systems, it was difficult to grasp the emotional state of new crew members in real time and provide appropriate feedback based on that. Therefore, when new crew members felt anxious or stressed, it was not possible to respond appropriately to their emotions, which led to a problem of reduced training effectiveness.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1108] In this invention, the server includes a means for recording the role-playing content of the new crew member, a means for transcribing the recorded content, a means for inputting the transcribed content into a sentiment analysis engine, a means for providing feedback based on the result of the sentiment analysis, and a means for simultaneously training the sales representative and the crew member. This makes it possible to grasp the emotional state of the new crew member in real time and provide appropriate feedback based on that.

[1109] The "means for recording the role-playing content of a new crew member" refers to a device or method for acquiring audio data of the role-playing performed by a new crew member.

[1110] A "means for transcribing recorded content" is a device or method for converting acquired voice data into text data.

[1111] "Means for inputting the transcribed content to a sentiment analysis engine" refers to an apparatus or method for transmitting text data to a sentiment analysis engine for analyzing the emotional state.

[1112] A "means for providing feedback based on the result of sentiment analysis" is a device or method for providing appropriate guidance or advice based on the emotional state result obtained from the sentiment analysis engine.

[1113] A "means for simultaneously developing sales representatives and crew" is a device or method for simultaneously educating or training both sales representatives and crew.

[1114] The system for implementing the present invention records the role-playing content of a new crew member, transcribes the recorded content, and performs sentiment analysis to provide appropriate feedback. A specific embodiment of this system will be described below.

[1115] 1. System Configuration

[1116] The system consists of the following main components:

[1117] Recording Equipment: Microphones for recording new crew members' role-playing in real time.

[1118] Speech recognition software: Speech recognition technology for converting recorded voice data into text information. Specifically, Google's speech recognition API and IBM's Watson are used.

[1119] Sentiment analysis engine: Software that receives text information as input and analyzes the emotional state. Specifically, it uses IBM Watson Tone Analyzer.

[1120] Feedback provision device: Displays and speakers for providing appropriate feedback to new crew members based on the results of sentiment analysis.

[1121] 2. Program Processing

[1122] The server first obtains the role-playing content of the new crew member using a recording device. Then, it converts the recorded content into text information using voice recognition software. After that, it inputs the text information into a sentiment analysis engine to analyze the emotional state. Finally, it provides appropriate feedback to the new crew member through a feedback providing device based on the result of the sentiment analysis.

[1123] 3. Hardware and Software Used

[1124] Hardware: Microphone, display, speakers

[1125] Software: Google speech recognition API, IBM Watson Tone Analyzer

[1126] 4. Specific Examples

[1127] For example, if a worker says, "This task is difficult," speech recognition will convert this into text information, and emotion analysis will detect "anxiety." As a result, the robot will provide feedback such as, "Don't worry, let's try a bit more slowly."

[1128] Examples of prompt statements

[1129] If a worker says, "This task is difficult," speech recognition converts this into text information, and emotion analysis detects "anxiety." As a result, the robot provides feedback such as, "Don't worry, let's try a little more slowly."

[1130] In this way, the system can grasp the emotional state of new crew members in real time and provide appropriate feedback based on that, which can improve the effectiveness of training new crew members.

[1131] The flow of the identification process in the application example 2 will be described with reference to FIG.

[1132] Step 1:

[1133] The server uses a microphone to record the role-playing content of the new crew member in real time, the input is the voice data of the new crew member, and the output is the recorded voice file, which is used in subsequent processing steps.

[1134] Step 2:

[1135] The server inputs the recorded voice file into speech recognition software and converts it into text information. Specifically, it uses Google's speech recognition API. The input is the voice file, and the output is text information (text data). This text data is used for sentiment analysis.

[1136] Step 3:

[1137] The server inputs the text information into a sentiment analysis engine and analyzes the emotional state. Specifically, it uses IBM Watson Tone Analyzer. The input is text information, and the output is the result of sentiment analysis (emotional state data). This emotional state data is used to provide feedback.

[1138] Step 4:

[1139] The server generates feedback based on the result of the sentiment analysis and provides it to the new crew member through the feedback providing device. Specifically, the feedback is displayed or audibly conveyed using a display or speaker. The input is the result of the sentiment analysis, and the output is a feedback message. For example, specific advice such as "Don't worry, try doing it a little more slowly" is provided.

[1140] Step 5:

[1141] The server monitors the effectiveness of the feedback and adjusts the feedback content as necessary. The input is the new crew's reaction and performance data, and the output is the adjusted feedback message. This allows the effectiveness of training new crew members to be continuously improved.

[1142] Example 3

[1143] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1144] Conventional sales training systems only recorded and transcribed the role-playing of new crew members, but were unable to reflect the user's emotions in the generated conversations and feedback. As a result, the feedback was one-sided, making it difficult to provide specific and useful advice that took into account the user's emotional reactions. In addition, there was a lack of means to simultaneously train salespeople and crew members.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1146] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for generating a conversation from the transcribed content using natural language processing technology, a means for systematizing the feedback content based on the generated conversation, a means for analyzing the user's emotions and reflecting the results in the feedback, and a means for simultaneously training the sales representative and the crew. This enables specific and useful feedback that takes into account the user's emotional reactions, and effectively trains the sales representative and the crew at the same time.

[1147] "Means for recording the role-playing content of newcomers" refers to devices or software used to record the audio of role-playing performed by new crew members.

[1148] "Means of transcribing the recording" refers to speech recognition technology or software used to convert recorded audio data into text data.

[1149] "Means of generating conversation from transcribed content using natural language processing technology" refers to natural language processing technology or generative AI models that use transcribed text data as input to generate natural conversation.

[1150] "Means of systematizing the feedback content based on the generated conversation" refers to technology and software for analyzing the generated conversation data, extracting important points and areas for improvement, and organizing and structuring the feedback content.

[1151] "Means of analyzing the user's emotions and reflecting the results in the feedback" refers to emotion recognition technology and software that infers emotions from the user's tone of voice, speaking style, choice of words, etc., and incorporates that information into the feedback content.

[1152] "Salesperson and crew development tools" refers to educational programs and systems designed to enable salespeople and new crew members to learn and grow at the same time.

[1153] This invention is a system that records the role-playing of new crew members, transcribes the recordings, generates natural conversations using a generative AI model, and organizes the feedback content. Furthermore, it analyzes the user's emotions and reflects the results in the feedback, providing more specific and useful feedback.

[1154] First, the user role-plays as a new crew member dealing with customers. The user uses the microphone of the terminal to record the conversation. For example, a recording application on a smartphone can be used to record the conversation.

[1155] The server then receives the recorded voice data and transcribes it using voice recognition technology. Specifically, it uses voice recognition software such as the Google Speech-to-Text API. The transcribed text data is then stored on the server.

[1156] The server then retrieves the transcribed text data and inputs it into a generative AI model, such as OpenAI GPT-4. The AI ​​generates natural conversation based on the input text and returns the results to the server.

[1157] The generated conversation data is then analyzed by the server, which uses natural language processing libraries such as NLTK and spaCy to extract important points and areas for improvement. Based on this information, the feedback is organized into a professional role for sales training.

[1158] In addition, the server inputs the recorded content into emotion recognition software to analyze the user's emotions. Specifically, it uses "IBM Watson Tone Analyzer" and other software. The server infers emotions from the user's tone of voice, speaking style, and choice of words, and reflects this information in the feedback.

[1159] For example, a new crew member role-plays a customer interaction and records the content. The recorded audio is transcribed using the Google Speech-to-Text API. The transcribed text is input into OpenAI GPT-4 to generate natural conversation. The generated conversation is analyzed using NLTK and spaCy, and the feedback content is organized as a professional role in sales training.

[1160] In addition, the system uses IBM Watson Tone Analyzer to analyze the user's emotions and reflects the results in the feedback. For example, if the user is nervous, the feedback will include that information, providing specific advice.

[1161] An example of a prompt sentence might be:

[1162] "A new crew member has role-played a customer interaction. Please transcribe the recording below and feed it into your generative AI model to generate natural-sounding conversations. Also, please use the emotion engine to analyze the user's emotions and incorporate the results into your feedback."

[1163] In this way, the system records the role-playing of the new crew member, transcribes it, generates a conversation, analyzes emotions, and finally organizes the feedback content. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1164] Step 1:

[1165] The user role-plays as a new crew member dealing with customers. The conversation is recorded using the device's microphone. The recorded voice data is input and saved in the device.

[1166] Step 2:

[1167] The server receives the recorded voice data from the device. The server uses "voice recognition technology" to convert the voice data into text data. Specifically, it calls the "voice recognition API" and uses the voice data as input to output text data. The converted text data is stored on the server.

[1168] Step 3:

[1169] The server retrieves the stored text data. The server uses the "generative AI model" to generate natural conversation using the text data as input. Specifically, it calls the "generative AI model API" and outputs natural conversation data using the text data as input. The generated conversation data is stored on the server.

[1170] Step 4:

[1171] The server obtains the generated conversation data. The server uses "natural language processing technology" to analyze the conversation data and extract important points and areas for improvement. Specifically, it uses a "natural language processing library" to input the conversation data and output feedback content. The feedback content is stored on the server.

[1172] Step 5:

[1173] The server analyzes the recorded content using "emotion recognition technology." Specifically, it calls the "emotion recognition API" and uses the recorded data as input to estimate the user's emotions. The emotion analysis results are stored on the server.

[1174] Step 6:

[1175] The server obtains the emotion analysis results and reflects them in the feedback content. Specifically, the server modifies the feedback content based on the emotion analysis results and provides specific advice that takes into account the user's emotional response. The final feedback content is provided to the user.

[1176] (Application example 3)

[1177] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1178] In the conventional training system for new operators, feedback was limited to general content and did not fully take into account the feelings and specific problems of individual operators. In addition, there was a lack of specific training on the operation of factory robots, making it difficult to provide efficient training.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the newcomer, a means for transcribing the recorded content, a means for inputting the transcribed content to the generative AI, a means for systematizing the feedback content as a professional role in sales training, a means for analyzing the user's emotions using an emotion engine and reflecting the results in the feedback content, and a means for educating new factory robot operators. This enables more specific and useful feedback that takes into account the emotions and specific problems of each operator.

[1180] "Means for recording the role-playing content of new recruits" refers to devices or software for recording the audio of role-playing performed by new operators.

[1181] A "means for transcribing recorded content" is a device or software that uses voice recognition technology to convert recorded voice data into text data.

[1182] "Means of inputting the transcribed content into a generative AI" refers to devices or software that input the transcribed text data into a chat generative AI and generate feedback in a natural conversational format.

[1183] The "means for systematizing feedback content from the perspective of a sales training professional" refers to a device or software for systematizing the generated feedback from the perspective of a sales training professional and organizing it as educational content.

[1184] "Means for analyzing a user's emotions using an emotion engine and reflecting the results in the feedback content" refers to a device or software that uses emotion recognition technology to analyze emotions from a user's voice or text and reflect the results in the feedback.

[1185] The "means for training new factory robot operators" refers to devices and software for training new operators on how to operate factory robots.

[1186] The system for carrying out the present invention is for efficiently training new operators. The system has the following configuration.

[1187] First, a new operator role-plays how to operate a factory robot. The content of this role-play is recorded in real time using a recording device. The recorded voice data is converted into text data using voice recognition technology. For example, the "speech_recognition" library is used for this voice recognition technology.

[1188] The transcribed text data is then fed into a chat generation AI, such as the GPT-3 model, which generates feedback in a natural conversational format based on the input text data.

[1189] Furthermore, the emotion engine is used to analyze the user's emotions. For example, the "emotion_recognition" library is used for this emotion engine. The emotion engine analyzes emotions from the user's voice and text, and reflects the results in the feedback.

[1190] Finally, the generated feedback is organized into a professional sales training roll, which contains specific advice and improvements that will help educate new operators.

[1191] As a concrete example, consider a scenario in which a new operator has a question about operating a factory robot. By inputting the following prompt sentences into the generative AI model, feedback can be obtained in a natural, conversational format.

[1192] Example prompt:

[1193] A new operator has a question about how to operate a factory robot. Generate feedback in a natural conversational format based on the text below.

[1194] Text: "I don't know how to operate this robot. What should I do?"

[1195] The system allows for more specific and useful feedback that takes into account each operator's feelings and specific problems, improving the efficiency of training new operators and helping them quickly acquire skills in operating factory robots.

[1196] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1197] Step 1:

[1198] The user role-plays the operation of a factory robot. The terminal records the voice of this role-play in real time using a recording device. The input is the user's voice, and the output is the recorded voice data.

[1199] Step 2:

[1200] The server converts the recorded voice data into text data using speech recognition technology. The "speech_recognition" library is used for this speech recognition technology. The input is the recorded voice data, and the output is the transcribed text data.

[1201] Step 3:

[1202] The server inputs the transcribed text data into a chat generation AI. The GPT-3 model is used for this chat generation AI. The input is the transcribed text data, and the output is feedback in a natural conversational format.

[1203] Step 4:

[1204] The server analyzes the user's emotions using an emotion engine, which uses the "emotion_recognition" library. The input is the transcribed text data, and the output is the result of the emotion analysis.

[1205] Step 5:

[1206] The server combines the generated feedback with the results of sentiment analysis and organizes the feedback content as a professional role in sales training. The input is the natural conversational feedback and the results of sentiment analysis, and the output is the organized feedback content.

[1207] Step 6:

[1208] The terminal provides the user with structured feedback content, where the input is the structured feedback content and the output is specific advice or improvements for the user.

[1209] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1210] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1211] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1212] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1213] [Third embodiment]

[1214] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1215] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1216] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1217] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1218] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1219] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1220] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1221] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1222] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1223] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1224] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1225] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1226] "Example 1"

[1227] In one embodiment of the present invention, a device is provided for recording the role-playing of new crew members. This device obtains voice data of the role-playing using, for example, a microphone or a recording function of a smartphone.

[1228] "Example 2"

[1229] Next, the acquired voice data is converted into text information. This conversion is performed, for example, using voice recognition technology. Specifically, it is possible to use voice recognition services such as Google's voice recognition API or IBM's Watson.

[1230] "Example 3"

[1231] The converted text information is then input into a generative AI, which is an AI that can mimic human conversation and generate natural conversation based on the input text information.

[1232] "Example 4"

[1233] Finally, the feedback content of the generated conversation is systematized as a professional role in sales training. This systematization can be done, for example, using text analysis and natural language processing technology.

[1234] The process flow of each embodiment will be described below.

[1235] "Example 1"

[1236] Step 1: The new crew member begins role-playing. This role-playing is based on a sales scenario, in which the new crew member simulates interactions with a customer.

[1237] Step 2: Once the role-playing session has started, record it using a microphone or smartphone recording function. This recording is done throughout the role-playing session, capturing in detail the comments and reactions of the new crew member.

[1238] "Example 2"

[1239] Step 1: Once the recording is complete, the audio data is converted into text information. This conversion is done using voice recognition technology. Specifically, it uses voice recognition services such as Google's voice recognition API or IBM's Watson.

[1240] Step 2: The speech recognition service converts the speech data into text data and outputs the result, which is used in the following steps.

[1241] "Example 3"

[1242] Step 1: The text data output from the speech recognition service is input into a generative AI, which is an AI that can imitate human conversation.

[1243] Step 2: The generative AI generates natural conversations based on the input text data. This generated conversation is used as feedback to new crew members.

[1244] "Example 4"

[1245] Step 1: Systematize the feedback content generated by the generative AI as a professional role in sales training. This systematization is done using text analysis and natural language processing technology.

[1246] Step 2: Structured feedback is provided to new crew members to help them improve their performance. This feedback also serves as a key input for developing both salespeople and crew members.

[1247] Example 1

[1248] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1249] In conventional role-playing for new crew members, evaluation and feedback of the recorded content was often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of consistency and objectivity in the evaluation, making effective training difficult. Furthermore, the recording data was not stored or analyzed sufficiently, and the data was not fully utilized.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1251] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for simultaneously training sales representatives and crew, a means for storing the recorded data in a cloud server, a means for analyzing the stored voice data, and a means for providing the evaluation results from the generative AI model to the user. This enables automatic analysis of the recorded content and provision of feedback, improving the consistency and objectivity of evaluation and realizing effective training. In addition, the recording data is efficiently stored and analyzed, promoting the utilization of data.

[1252] "Means for recording the role-playing content of new crew members" refers to a device or function for acquiring audio of the role-playing performed by new crew members.

[1253] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[1254] "Means of inputting the transcribed content into the generative AI model" refers to the process or function by which the transcribed text data is passed to the generative AI model.

[1255] The "means of systematizing feedback content for the role of sales training professionals" refers to a method or system for organizing and systematizing the feedback provided by sales training experts based on the evaluation results obtained from the generative AI model.

[1256] A "means for simultaneously training sales representatives and crew" is a method or system for simultaneously educating and training both sales representatives and new crew members.

[1257] "Means for storing recorded data on a cloud server" refers to the technology and process for storing acquired voice data on a cloud server via the Internet.

[1258] "Means for analyzing stored voice data" refers to technology or a system for analyzing voice data stored on a cloud server and generating evaluations and feedback.

[1259] "Means for providing evaluation results from a generative AI model to a user" refers to a method or system for communicating the evaluation results generated by the generative AI model to a user.

[1260] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1261] The present invention relates to a system for recording the role-playing of a new crew member, analyzing the content of the recording, and providing feedback. A specific embodiment of this system will be described below.

[1262] System Configuration

[1263] This system is configured using the following hardware and software.

[1264] 1. Hardware:

[1265] Microphone: Used to capture audio for roleplaying of new crew members.

[1266] Smartphone: Use the built-in recording function to capture audio data.

[1267] 2. Software:

[1268] Recording application: Record audio data using a recording application installed on your smartphone.

[1269] Data storage system: The acquired voice data is stored on a cloud server.

[1270] Generative AI model: Analyzes captured audio data and generates role-playing evaluations and feedback.

[1271] System Operation

[1272] Acquiring audio data:

[1273] Before starting role-playing, the user launches the recording application on the smartphone. The device (smartphone) picks up the role-playing audio through the microphone and starts recording. The user performs role-playing and taps the "stop recording" button when finished.

[1274] Audio data storage:

[1275] After the device finishes recording, it uploads the acquired voice data to the cloud server, and the server stores the uploaded voice data in the data storage system.

[1276] Analysis of audio data:

[1277] The server inputs the saved voice data into the generative AI model for analysis. The generative AI model analyzes the voice data and evaluates it on the evaluation items of role-playing (clarity of pronunciation, flow of speech, naturalness of dialogue, and overall evaluation).

[1278] Generate feedback:

[1279] The server receives the evaluation results from the generative AI model and generates feedback to provide to the user. The server sends the generated feedback to the user's device. The user checks the feedback on the device and understands what needs to be improved.

[1280] Examples

[1281] As a concrete example, the following prompt sentence is input into the generative AI model.

[1282] Example prompt:

[1283] "Please analyze the audio data of the new crew member's role-playing and rate it on the following items: 1. Clarity of pronunciation 2. Flow of speech 3. Naturalness of dialogue 4. Overall rating."

[1284] By inputting this prompt sentence into the generative AI model, the model analyzes the voice data and outputs an evaluation for each item. For example, the generative AI model generates feedback like the following:

[1285] Feedback example:

[1286] "Clarity of pronunciation: 8 / 10, flow of speech: 7 / 10, naturalness of dialogue: 9 / 10, overall rating: 8 / 10. Pronunciation was generally clear, but some words were hard to hear. Flow of speech was good, but could have been smoother. Naturalness of dialogue was very high, and overall it was good role-playing."

[1287] In this way, a system is realized in which the server, terminals, and users cooperate to record and evaluate the role-playing of new crew members.

[1288] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1289] Step 1: Prepare to record

[1290] Before starting role-playing, the user launches the recording application on the smartphone. The device displays a pop-up message notifying the completion of the application launch. The input is the user's operation, and the output is the notification that the recording application has been launched.

[1291] Step 2: Acquire audio data

[1292] The user taps the "Start Recording" button on the recording application. The device picks up the role-playing voice in real time through the microphone and starts recording. The user performs the role-playing and taps the "Stop Recording" button when finished. The input is the user's voice, and the output is the recorded voice data.

[1293] Step 3: Save the audio data

[1294] The device temporarily stores the audio data after recording in its internal storage. The device then uploads the stored audio data to a cloud server. The server receives the uploaded audio data and stores it in a data storage system. The input is the recorded audio data, and the output is the audio data stored in the cloud server.

[1295] Step 4: Analyze the audio data

[1296] The server inputs the stored voice data into the generative AI model. The generative AI model analyzes the voice data and evaluates it on the evaluation items of role-playing (clarity of pronunciation, flow of story, naturalness of dialogue, and overall evaluation). The input is the voice data stored on the cloud server, and the output is the evaluation result.

[1297] Step 5: Generate feedback

[1298] The server receives the evaluation results from the generative AI model and generates feedback to provide to the user. The server sends the generated feedback to the user's device. The user checks the feedback on the device and understands future improvements. The input is the evaluation results from the generative AI model, and the output is the feedback provided to the user.

[1299] (Application example 1)

[1300] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1301] It is necessary to provide effective feedback and train new crew members quickly and efficiently in role-playing. However, with conventional methods, analyzing the recorded content and providing feedback is time-consuming and difficult, making it difficult to train efficiently.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1303] In this invention, the server includes a means for recording the role-playing of newcomers, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for simultaneously training sales representatives and crew members, a means for providing feedback from the generative AI model to new crew members, and a means for pointing out areas for improvement based on the feedback. This allows for quick and effective feedback to be provided to the role-playing of new crew members, enabling efficient training.

[1304] "Means for recording the role-playing content of new crew members" refers to a device or function for acquiring audio of the role-playing performed by new crew members.

[1305] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[1306] "Means for inputting the transcribed content into the generative AI model" refers to means for sending transcribed text data to the generative AI model for analysis and feedback.

[1307] "Means of systematizing feedback content from the perspective of a sales training professional" refers to a means of organizing the feedback obtained from the generative AI model and systematizing it from the perspective of a sales training professional.

[1308] "Means for training sales representatives and crew members simultaneously" refers to means for training both sales representatives and new crew members simultaneously.

[1309] "Means for providing feedback from the generative AI model to new crew members" refers to means for providing feedback to new crew members based on the results of analysis by the generative AI model.

[1310] "Means for pointing out areas for improvement based on feedback" refers to means for specifically pointing out areas for improvement for new crew members based on feedback from the generative AI model.

[1311] The system for implementing the present invention includes a series of processes for recording the role-playing content of a new crew member, transcribing the recorded content, inputting the content into a generative AI model, and providing feedback. A specific embodiment of the system will be described below.

[1312] Hardware and Software Configuration

[1313] The system uses the following hardware and software:

[1314] Hardware: Smartphone microphone

[1315] Software: Python, sounddevice library (audio recording), requests library (HTTP requests)

[1316] Data processing and calculation

[1317] 1. Recording:

[1318] The role-playing of new crew members is recorded in real time using a smartphone microphone.

[1319] The recorded audio data is obtained as a numpy array.

[1320] 2. Transcription:

[1321] The recorded voice data is converted into text information using voice recognition technology.

[1322] The transcribed text data is fed into a generative AI model.

[1323] 3. Analysis using generative AI models:

[1324] The transcribed text data is sent to a generative AI model for analysis.

[1325] The generative AI model analyzes the audio data and generates feedback.

[1326] 4. Providing Feedback:

[1327] The feedback obtained from the generative AI model is provided to new crew members.

[1328] Based on the feedback, we will provide specific areas for improvement for new crew members.

[1329] Examples

[1330] New crew members role-play customer service and record their voices. The recorded data is sent to a generative AI model, which provides feedback such as "your voice tone is too low" or "your greeting to customers is insufficient." This allows new crew members to identify specific areas for improvement and efficiently improve their skills.

[1331] Examples of prompt statements

[1332] Analyze the role-playing audio of the new crew member and provide feedback on the following:

[1333] 1. Tone of voice

[1334] 2. Speaking Speed

[1335] 3. Greeting customers

[1336] 4. Responding to Questions

[1337] In this way, training of new staff members in physical stores can be made more efficient and high quality customer service can be achieved.

[1338] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1339] Step 1:

[1340] The terminal (smartphone) records the role-playing content of the new crew member. The recording is done using the smartphone's microphone, and the audio data is obtained as a numpy array. The input is the new crew member's voice, and the output is the recorded audio data.

[1341] Step 2:

[1342] The device converts the recorded voice data into text information using voice recognition technology. Specifically, the voice recognition software analyzes the voice data and generates corresponding text data. The input is the voice data, and the output is the transcribed text data.

[1343] Step 3:

[1344] The terminal sends the transcribed text data to the generative AI model. Specifically, the terminal sends the text data as an HTTP request to the API of the generative AI model. The input is the transcribed text data, and the output is the request sent to the generative AI model.

[1345] Step 4:

[1346] The server (generative AI model) analyzes the received text data and generates feedback. The generative AI model evaluates the text data based on the prompt sentence and generates specific feedback. The input is the text data and the output is the feedback content.

[1347] Step 5:

[1348] The server sends the generated feedback back to the terminal. Specifically, it sends the feedback content to the terminal as an HTTP response. The input is the feedback content from the generative AI model, and the output is the feedback sent to the terminal.

[1349] Step 6:

[1350] The terminal provides the received feedback to the new crew member. Specifically, the feedback content is displayed on a display screen so that the new crew member can check it. The input is the feedback content from the server, and the output is the feedback provided to the new crew member.

[1351] Step 7:

[1352] The user (new crew member) identifies areas for improvement based on the provided feedback and reflects them in the next role-playing. Specifically, the user checks the feedback and performs the role-playing while keeping in mind the areas for improvement pointed out. The input is the feedback, and the output is the improved role-playing.

[1353] Example 2

[1354] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1355] In conventional systems that convert voice data into text information, the process from acquiring voice data to converting it into text information and then using it was complicated, making it difficult to carry out efficiently. There were also issues with the accuracy of converting voice data into text information using voice recognition technology, and with the management and use of the converted data. This caused problems with the time and effort required when users tried to use it as a prompt to input into a generative AI model.

[1356] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice data, a means for transmitting the acquired voice data to the server, a means for converting the voice data into character information, a means for storing the converted character information in a database, and a means for inputting the stored character information into a generative AI model. This makes it possible to efficiently perform the process from acquiring voice data to converting it into character information and using it.

[1357] "Audio data" refers to data in which audio is recorded in digital format.

[1358] "Means of acquisition" refers to devices and software for collecting voice data.

[1359] "Transmitting means" refers to a communication means for transferring the acquired voice data to a server.

[1360] "Means for converting into text information" refers to technology that analyzes voice data and generates corresponding text data.

[1361] "Means of storing in a database" refers to a system for systematically storing the converted character information.

[1362] "Means of input to the generative AI model" refers to the interfaces or processes used to provide the stored textual information to the AI ​​model.

[1363] "Means of recording in real time" refers to devices or software for instantly recording audio as digital data.

[1364] "Speech recognition technology" refers to technology for analyzing voice data and generating corresponding text information.

[1365] The present invention is a system for efficiently converting voice data into text information and inputting the text information into a generative AI model. A specific embodiment of the system will be described below.

[1366] First, a user uses a terminal for acquiring voice data. The terminal includes a voice input device such as a smartphone and a microphone. The user uses these devices to record the voice data. For example, when recording the contents of a meeting, the user can use a recording app on the smartphone.

[1367] The device then transmits the captured voice data to a server over the Internet, using HTTP requests to upload the voice data to the server, which then runs software to process the received voice data.

[1368] The server converts the received voice data into text information. This conversion is performed using voice recognition technology. Specifically, a voice recognition API or voice recognition service can be used. For example, Google's voice recognition API or IBM's Watson can be used. The server sends the voice data to these services and receives the text information in return.

[1369] The server then stores the converted text in a database, along with other metadata such as user ID and timestamp, to make it easier for users to search for the data later.

[1370] Finally, the user uses the saved text information as a prompt to be input to the generative AI model. The user opens a web browser and logs into the system's web interface. The user accesses the "Meeting Records" page and checks the saved text information. If necessary, the user copies the text information and uses it as a prompt to be input to the generative AI model.

[1371] As a concrete example, consider a case where a user wants to record the audio of a meeting and save the content as text information. The user uses a smartphone to record the audio of the meeting and sends the audio data to a server. The server uses a speech recognition API to convert the audio data into text information and stores the text information in a database. Later, the user can use the text information as a prompt sentence to input into a generative AI model.

[1372] An example of a prompt might be:

[1373] "Please summarize the following meeting: 'In today's meeting, we discussed the launch date of the new product. The marketing team proposed to set the launch date on the 15th of next month, but the development team said that it is not ready yet.'"

[1374] In this way, users can efficiently convert voice data into textual information that can be used in generative AI models.

[1375] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1376] Step 1:

[1377] A user obtains audio data using a smartphone or microphone. Specifically, the user opens a recording app on the smartphone and taps the "Start Recording" button to record the audio of the meeting. When recording is finished, the user taps the "Stop Recording" button to save the audio data. The input is audio, and the output is digital audio data.

[1378] Step 2:

[1379] The terminal transmits the acquired voice data to the server. Specifically, the user selects the saved voice data from the recording app and taps the "Send" button. This uploads the voice data to the server via an HTTP request. The input is the digital voice data, and the output is the voice data transmitted to the server.

[1380] Step 3:

[1381] The server converts the received voice data into text information. Specifically, the server sends the voice data to a voice recognition API, which analyzes the voice data and generates corresponding text information. For example, the server sends voice data to a voice recognition service and receives the returned text information. The input is voice data and the output is text information.

[1382] Step 4:

[1383] The server saves the converted text information in a database. Specifically, the server inserts the received text information into the "meeting records" table in the database. At this time, metadata such as the user ID and timestamp are also saved. The input is text information and metadata, and the output is the text information saved in the database.

[1384] Step 5:

[1385] The user uses the saved text information as a prompt sentence to be input to the generative AI model. Specifically, the user opens a web browser and logs into the system's web interface. The user accesses the "Meeting Records" page and checks the saved text information. If necessary, the user copies this text information and uses it as a prompt sentence to be input to the generative AI model. The input is the text information obtained from the database, and the output is the prompt sentence to be input to the generative AI model.

[1386] (Application example 2)

[1387] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1388] In conventional food delivery services, when delivery personnel manually input order details from customers, input errors and time loss were an issue. In addition, delivery personnel had to use their hands to input the order details, which made it difficult to carry out work efficiently.

[1389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for acquiring voice data, a means for converting the acquired voice data into text information, and a means for registering the converted text information in the order system. This makes it possible to efficiently process the order details input by voice by the delivery person.

[1390] "Audio data" refers to data in which audio is recorded in digital format.

[1391] "Means for acquiring" refers to a device or method for collecting audio data.

[1392] "Text information" refers to information obtained by converting voice data into text.

[1393] "Means of converting" refers to technology or equipment for converting voice data into text information.

[1394] "Order System" refers to a system for managing and processing orders from customers.

[1395] "Means for registering" refers to a method or device for inputting and storing converted character information into the ordering system.

[1396] "Management Means" means the methods and devices used to organize, track and manage orders.

[1397] A "delivery person" refers to a person whose role is to deliver goods to customers.

[1398] "Mobile device" refers to an electronic device that can be carried by a delivery person.

[1399] "Voice recognition technology" refers to the technology that analyzes voice data and converts it into text information.

[1400] To implement this invention, you need a mobile terminal for acquiring voice data, voice recognition technology for converting the voice data into text information, and a means for registering the converted text information in the ordering system.

[1401] System Configuration

[1402] 1. Mobile terminal: A mobile terminal such as a smartphone or smart glasses is used by the delivery person. This allows the delivery person to input the order details from the customer by voice.

[1403] 2. Speech recognition technology: To convert voice data into text information, we use voice recognition services such as Google's speech recognition API and IBM's Watson, which allows us to convert voice data into text information with high accuracy.

[1404] 3. Ordering system: An ordering system is used to manage and process the converted text information. This system has the function of automatically registering and managing the order contents.

[1405] Program Processing

[1406] The server first receives voice data from the mobile device. It then converts the voice data into text using voice recognition technology. The converted text is automatically registered in the order system. This allows delivery personnel to input order details without using their hands, making work more efficient.

[1407] Examples

[1408] For example, if a delivery person voice-inputs "one pizza, two colas," the mobile device captures this voice data. Next, the voice data is converted into text information using voice recognition technology, and registered in the ordering system as "one pizza, two colas." This series of processes allows the delivery person to quickly and accurately register the order details in the system.

[1409] Examples of prompt statements

[1410] "Please convert the following audio data into text: 1 pizza, 2 cokes"

[1411] In this way, a series of processes can be realized, from converting voice data into text information and registering it in the order system. This system improves the work efficiency of delivery personnel and reduces the risk of input errors.

[1412] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1413] Step 1:

[1414] The user (delivery person) inputs voice data using a mobile terminal. Specifically, the delivery person speaks the order details into a smartphone or smart glasses. The input is recorded as voice data on the mobile terminal.

[1415] Step 2:

[1416] The terminal acquires the voice data. The acquired voice data is temporarily stored in the memory of the terminal. The input is the delivery person's voice data, and the output is the voice data stored in the terminal.

[1417] Step 3:

[1418] The terminal transmits the acquired voice data to the server. Specifically, the terminal uploads the voice data to the server via the Internet. The input is the voice data stored in the terminal, and the output is the voice data transmitted to the server.

[1419] Step 4:

[1420] The server receives the voice data and converts it into text information using voice recognition technology. Specifically, the server calls Google's voice recognition API or IBM's Watson to convert the voice data into text information. The input is the voice data sent to the server, and the output is the converted text information.

[1421] Step 5:

[1422] The server registers the converted text information in the ordering system. Specifically, the server saves the text information in the ordering system's database and registers it as the order details. The input is the converted text information, and the output is the order details registered in the ordering system.

[1423] Step 6:

[1424] The ordering system manages the registered order details. Specifically, the ordering system organizes the order details and sends notifications to delivery personnel and customers as necessary. The input is the order details registered in the ordering system, and the output is the managed order information.

[1425] Step 7:

[1426] The user (delivery person) receives a notification from the ordering system and confirms the order details. Specifically, the delivery person receives a notification from the ordering system through a mobile terminal and confirms the order details. The input is the notification from the ordering system, and the output is the order details confirmed by the delivery person.

[1427] Example 3

[1428] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1429] In conventional sales training systems, it was difficult to effectively provide feedback and systematize the role-playing content of new recruits. In addition, the quality of the generated conversations and the accuracy of the feedback were low, making it inefficient to simultaneously train sales representatives and crew members. This limited the effectiveness of the training and made it difficult to improve practical skills.

[1430] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1431] In this invention, the server includes a means for recording the role-playing content of the new recruit, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for receiving the generated conversation, a means for systematizing the feedback content of the generated conversation using text analysis and natural language processing technology, and a means for simultaneously training the sales representative and the crew. This improves the quality of the generated conversation, increases the accuracy of the feedback, and makes it possible to efficiently train the sales representative and the crew at the same time.

[1432] "Recording means" refers to a device or software that records the role-playing content of a newcomer as audio data.

[1433] A "transcription means" is a technology or device that converts recorded audio data into text data.

[1434] A "generative AI model" is an artificial intelligence algorithm or system that generates natural-sounding conversations based on input text information.

[1435] A "means for receiving the generated conversation" is a device or software that obtains the conversation data output from the generative AI model.

[1436] "Text analysis" is a technique or method for analyzing text data and understanding its meaning and structure.

[1437] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[1438] A "means for systematizing feedback content" is a technique or device for structuring and organizing feedback based on the generated conversation as a professional role in sales training.

[1439] A "means for simultaneously training salespeople and crew" is a system or method for effectively training salespeople and new crew members simultaneously using generated conversations and feedback.

[1440] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1441] This invention is a system that records the role-playing content of new recruits, transcribes the recording content, inputs it into a generative AI model, and organizes the generated conversation as feedback. A specific embodiment of this system will be described below.

[1442] Server Roles

[1443] The server receives text information entered by the user. Specifically, the server receives an HTTP request and extracts text information from the request body. Next, the server inputs the received text information into a generative AI model. For example, OpenAI's GPT-4 can be used as this generative AI model. The server sends the text information to the generative AI model as an API request and receives the generated conversation. The generated conversation is analyzed using text analysis and natural language processing technology, and the feedback content is systematized. For this analysis, Python's NLTK library or SpaCy can be used.

[1444] Terminal Roles

[1445] The terminal transmits the character information input by the user to the server. Specifically, the terminal transmits the character information input by the user to the server as an HTTP request. After receiving the generated conversation from the server, the terminal displays the conversation to the user. Furthermore, the terminal provides an interface for the user to input feedback. This allows the user to input feedback for the generated conversation.

[1446] User Roles

[1447] The user inputs text information using the terminal. Specifically, the user inputs text information using a keyboard or a touch screen of the terminal. The user checks the generated conversation on the terminal and inputs feedback as necessary. The user inputs feedback using a feedback input interface of the terminal and checks that the content is organized.

[1448] Examples

[1449] For example, in a sales training scenario, a user (sales trainee) types into the terminal, "Please explain the features of the new product." The terminal sends this text information to the server, which then sends the text information to GPT-4 as an API request. GPT-4 generates a dialogue such as, "The features of the new product are that it is lightweight, easy to carry, and has a long battery life." The server receives the generated dialogue and systematizes the feedback content using text analysis and natural language processing techniques. The terminal displays the generated dialogue and feedback content to the user, who then checks it and enters feedback.

[1450] Examples of prompt statements

[1451] Prompt: "Describe the features of your new product."

[1452] When this prompt sentence is input to GPT-4, the AI ​​generates a natural conversation and provides useful feedback for the sales training scenario. The flow of the specific process in Example 3 is explained with reference to FIG.

[1453] Program processing flow

[1454] Step 1: Enter text information

[1455] Subject: User

[1456] The user uses the terminal to input text information. Specifically, the user uses the keyboard or touch screen of the terminal to input text information such as "Please explain the features of the new product." This input text information becomes the input data for the next step.

[1457] Step 2: Send text information

[1458] Subject: Terminal

[1459] The terminal sends the character information entered by the user to the server. Specifically, the terminal generates an HTTP request, includes the character information in the request body, and sends it to the server. The input of this step is the character information entered by the user, and the output is the HTTP request sent to the server.

[1460] Step 3: Receiving text information

[1461] Subject: Server

[1462] The server receives the HTTP request sent from the terminal and extracts text information from the request body. Specifically, the server analyzes the HTTP request and obtains the text information. The input of this step is the HTTP request from the terminal, and the output is the extracted text information.

[1463] Step 4: Inputting text information into the AI ​​model

[1464] Subject: Server

[1465] The server inputs the extracted character information into the generative AI model. Specifically, the server generates an API request to the generative AI model (e.g., GPT-4) and sends the character information. The input of this step is the extracted character information, and the output is an API request to the generative AI model.

[1466] Step 5: Receive the generated conversation

[1467] Subject: Server

[1468] The server receives the generated conversation from the generative AI model. Specifically, the server receives the API response from the generative AI model and extracts the generated conversation. The input of this step is the API response from the generative AI model, and the output is the generated conversation.

[1469] Step 6: Organize your feedback

[1470] Subject: Server

[1471] The server organizes the feedback content of the generated conversation using text analysis and natural language processing techniques. Specifically, the server analyzes the generated conversation and organizes the feedback content using Python's NLTK library and SpaCy. The input of this step is the generated conversation, and the output is the organized feedback content.

[1472] Step 7: Receive the generated conversation

[1473] Subject: Terminal

[1474] The terminal receives the generated conversation from the server. Specifically, the terminal receives the HTTP response from the server and extracts the generated conversation. The input of this step is the HTTP response from the server, and the output is the generated conversation.

[1475] Step 8: View the generated conversation

[1476] Subject: Terminal

[1477] The terminal displays the generated conversation to the user. Specifically, the terminal displays the generated conversation on a user interface. The input of this step is the generated conversation, and the output is the conversation displayed to the user.

[1478] Step 9: Provide feedback

[1479] Subject: User

[1480] The user inputs feedback for the generated conversation. Specifically, the user inputs feedback through a feedback input interface of the terminal. The input of this step is the generated conversation, and the output is the feedback input by the user.

[1481] Step 10: Review the feedback

[1482] Subject: User

[1483] The user confirms that the feedback content is organized. Specifically, the user confirms the result of the feedback content organization displayed on the terminal screen. The input of this step is the organized feedback content, and the output is the confirmed feedback content.

[1484] (Application example 3)

[1485] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1486] Conventional sales training systems lacked a means to effectively provide feedback on the role-playing content of new employees, which led to slow improvement in sales skills. In addition, because the feedback content was not systematized, it was difficult to provide appropriate guidance to individual sales representatives. Furthermore, there was a need for an efficient training method that utilizes smartphones.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1488] In this invention, the server includes a means for recording the role-playing content of new recruits, a means for transcribing the recorded content, a means for inputting the transcribed content into a generative AI, a means for systematizing the feedback content as a professional role of sales training, a means for simultaneously training sales representatives and crew members, a means for analyzing the generated conversation using text analysis and natural language processing technology and providing feedback, and a means including an application to be installed on a smartphone. This makes it possible to improve the efficiency of sales training and provide effective feedback.

[1489] "Means for recording the role-playing content of new recruits" refers to devices or software for recording the audio of the role-playing performed by new recruits.

[1490] A "means for transcribing recorded content" refers to technology or equipment for converting recorded audio data into text data.

[1491] "Means of inputting the transcribed content into the generative AI" refers to the interface or process for inputting the transcribed text data into the chat generative AI.

[1492] "Means of systematizing feedback content as a sales training professional" refers to techniques and methods for analyzing the generated conversation content from the perspective of a sales training professional and structuring the feedback.

[1493] A "means of simultaneous salesperson and crew development" is a system or method by which salespeople and new crew members learn and improve their skills at the same time.

[1494] "Means for analyzing the generated conversation using text analysis and natural language processing technology and providing feedback" refers to technologies and methods for evaluating the content of the generated conversation using text analysis and natural language processing technology and providing feedback based on the results.

[1495] "Means including an application installed on a smartphone" refers to a system or software including an application that runs on a smartphone.

[1496] A system for implementing this invention has the following configuration. First, a smartphone or a dedicated recording device is used as a means for recording the role-playing content of the new recruit. The recorded voice data is converted into text information using voice recognition technology. Google Speech-to-Text API, IBM Watson Speech to Text, etc. can be used for this voice recognition technology.

[1497] Next, the transcribed text data is input into a chat generation AI. For this chat generation AI, OpenAI's GPT-3 is used. GPT-3 can generate natural conversations based on the input text data.

[1498] The generated conversation content is analyzed using text analysis and natural language processing technology. Specifically, libraries such as TextBlob and NLTK (Natural Language Toolkit) are used to evaluate the positivity and subjectivity of the conversation. Based on the results of this analysis, feedback is provided from the perspective of sales training professionals.

[1499] This feedback is provided through an application installed on a smartphone. The application provides an interface for salespeople and new crew members to simultaneously learn and improve their skills. For example, it displays the conversations generated and their feedback, and allows users to evaluate their own performance.

[1500] As a concrete example, if a salesperson were simulating a conversation with a customer looking for a new smartphone, they might use the following prompt:

[1501] "Customer: Hello, I'm looking for a new smartphone today. Which model would you recommend?"

[1502] If we feed this prompt into a generative AI model, it might generate the following dialogue:

[1503] “Customer: Hello, I’m looking for a new smartphone today. Which model would you recommend?

[1504] Staff: Hello, customer. I understand you are looking for a new smartphone. We can recommend some models based on your needs. For example, the latest model is very popular and has a great camera. We can also recommend other high-performance models. What features do you care about?

[1505] The conversation is analyzed by TextBlob and feedback is provided. For example, feedback such as "Positiveness: 0.5, Subjectivity: 0.3" can be obtained. Based on this feedback, sales staff can improve their responses.

[1506] As described above, the present invention realizes efficient sales training and effective feedback provision by combining the means of recording, transcription, chat generation, text analysis, and feedback provision.

[1507] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1508] Step 1:

[1509] A user starts role-playing using a smartphone.

[1510] Input: User's voice

[1511] Output: Recorded audio data

[1512] Specific operation: The user activates the recording function of the smartphone and starts role-playing. The smartphone records the user's voice.

[1513] Step 2:

[1514] The server receives the recorded voice data and converts it into text information using voice recognition technology.

[1515] Input: Recorded audio data

[1516] Output: Transcribed text data

[1517] Specific operation: The server uses the Google Speech-to-Text API or IBM Watson Speech to Text to convert the voice data into text data.

[1518] Step 3:

[1519] The server inputs the transcribed text data into a chat generation AI, which generates natural conversation.

[1520] Input: Transcribed text data

[1521] Output: Generated conversation text

[1522] How it works: The server uses OpenAI's GPT-3 to generate natural-sounding conversations based on input text data.

[1523] Step 4:

[1524] The server analyzes the generated conversation text using text analysis and natural language processing techniques and provides feedback.

[1525] Input: Generated conversation text

[1526] Output: Feedback data (positivity, subjectivity, etc.)

[1527] Specific operation: The server uses TextBlob and NLTK to analyze the generated conversation text and evaluate its positivity and subjectivity.

[1528] Step 5:

[1529] The server sends the feedback data to the smartphone and displays it to the user.

[1530] Input: Feedback data

[1531] Output: Feedback information displayed to the user

[1532] Specific operation: The server sends the feedback data to the smartphone, and the smartphone application displays the feedback information to the user.

[1533] Step 6:

[1534] Users can view feedback and use it to improve their sales skills.

[1535] Input: Feedback information displayed to the user

[1536] Output: Understanding users and improving their behavior

[1537] Specific Actions: Users review the feedback information displayed on their smartphone screen, evaluate their sales skills, and identify areas for improvement.

[1538] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1539] "Example 1"

[1540] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system records the role-playing of a new crew member and transcribes the recorded content. The transcribed content is input to a generative AI, which organizes the feedback content as a professional role in sales training. Furthermore, this system includes an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from the recorded content and reflects the results in the feedback content. Specifically, the emotion is estimated from the user's tone of voice, speaking style, choice of words, etc., and the information is used in the feedback. This enables more specific and useful feedback that also takes into account the emotional reactions of the new crew member.

[1541] "Example 2"

[1542] The emotion engine also optimizes the training of salespeople and crew members based on the results of emotion analysis. Specifically, it analyzes how new crew members emotionally react to certain scenarios and how those reactions affect sales performance, and adjusts the training program accordingly. For example, if it senses that new crew members are anxious about tough questions from customers, it can provide training to strengthen their ability to handle such scenarios.

[1543] "Example 3"

[1544] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system records the role-playing of a new crew member and transcribes the recorded content. The transcribed content is input to a generative AI, which organizes the feedback content as a professional role in sales training. Furthermore, this system includes an emotion engine that recognizes the user's emotions. This emotion engine analyzes the user's emotions from the recorded content and reflects the results in the feedback content. Specifically, the emotion is estimated from the user's tone of voice, speaking style, choice of words, etc., and the information is used in the feedback. This enables more specific and useful feedback that also takes into account the emotional reactions of the new crew member.

[1545] The process flow of each embodiment will be described below.

[1546] "Example 1"

[1547] Step 1: Have new crew members role-play and record the session.

[1548] Step 2: Transcribe the recorded session.

[1549] Step 3: Input the transcribed content into the generative AI.

[1550] Step 4: The generative AI generates the feedback content.

[1551] Step 5: The emotion engine analyzes the user's emotions from the recording.

[1552] Step 6: The emotion engine applies the analysis results to the feedback content.

[1553] "Example 2"

[1554] Step 1: The emotion engine analyzes the user's emotions from the recording.

[1555] Step 2: The sentiment engine uses the results of sentiment analysis to optimize the training of sales representatives and crew.

[1556] Step 3: The emotion engine analyzes how the new crew member will emotionally react to a particular scenario.

[1557] Step 4: The emotion engine analyzes how that response impacts sales outcomes.

[1558] Step 5: The emotion engine adjusts the training program based on the analysis results.

[1559] Example 1

[1560] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1561] In the past, role-playing for new crew members required the manual process of transcribing recordings and providing feedback, which was time-consuming and labor-intensive. In addition, the feedback did not take into account emotional reactions, limiting the improvement of response skills in actual work. Furthermore, there was a lack of effective means to simultaneously train salespeople and crew members.

[1562] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the newcomer, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for performing emotion analysis using an emotion engine that recognizes the user's emotions, a means for reflecting the emotion analysis result in the feedback content, and a means for simultaneously training the salesperson and the crew. This makes it possible to automate the process from recording to providing feedback and provide specific and useful feedback that takes into account emotional reactions. In addition, it is possible to effectively train the salesperson and the crew at the same time.

[1563] "Means for recording the role-playing content of new crew members" refers to devices or functions for recording the audio of the role-playing performed by new crew members.

[1564] A "means of transcribing the recorded content" refers to technology or software for converting recorded audio data into text data.

[1565] "Means for inputting the transcribed content into the generative AI model" refers to the process of sending the transcribed text data to the generative AI model to generate feedback content.

[1566] "A means of systematizing feedback content for the role of a sales training professional" refers to a method that uses a generative AI model to structure and provide feedback like that given by a sales training professional.

[1567] The "means for performing emotion analysis using an emotion engine that recognizes the user's emotions" refers to software or algorithms for analyzing emotions from a user's voice or text.

[1568] The "means for reflecting the emotion analysis result in the feedback content" is a method for incorporating the emotion analysis result obtained by the emotion engine into the generated feedback content.

[1569] A "means of simultaneously developing salespeople and crew" is an educational program or system that allows salespeople and new crew members to learn and grow at the same time.

[1570] This invention is a system that effectively records the role-playing of new crew members, transcribes the contents, and provides feedback. Furthermore, by analyzing the user's emotions and reflecting the results in the feedback, it is possible to provide more specific and useful feedback.

[1571] Hardware and Software Used

[1572] Recording device

[1573] When the user role-plays a new crew member, the user can obtain voice data using a microphone or the recording function of the smartphone. For example, the user can use a voice memo app on the smartphone or a dedicated recording device.

[1574] Speech Recognition Software

[1575] The server uses speech recognition software to transcribe the captured audio data, specifically using a speech recognition service such as the Google Cloud Speech-to-Text API. This process converts the audio data into text data.

[1576] Generative AI Models

[1577] The server inputs the transcribed text data into a generative AI model, such as OpenAI's GPT-3, which then organizes the feedback into a professional role in sales training.

[1578] Sentiment Analysis Tools

[1579] The server uses an emotion engine to analyze the user's emotions from the recordings, using emotion analysis tools such as IBM Watson Tone Analyzer, which infers emotions from the user's tone of voice, speaking style, and choice of words.

[1580] Examples

[1581] New crew members role-play customer service and record the content on their smartphones. The recorded voice data is sent to a server and transcribed using Google Cloud Speech-to-Text API. The transcribed text data is then input into GPT-3, which generates feedback content as a professional role in sales training. In addition, sentiment analysis is performed using IBM Watson Tone Analyzer, and the results are reflected in the feedback.

[1582] Examples of prompt statements

[1583] "A new crew member has role-played interacting with a customer. Please transcribe the audio data below and generate feedback. Also, please perform sentiment analysis and incorporate the results into your feedback."

[1584] In this way, users can get specific, useful feedback on their role-playing of new crew members that also takes into account their emotional reactions.

[1585] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1586] Step 1:

[1587] When a user role-plays a new crew member, the user acquires voice data using a microphone or the recording function of the smartphone. Specifically, the user opens a voice memo app on the smartphone and presses the "Start Recording" button to record the role-playing. When the recording is finished, the user presses the "Stop Recording" button to save the voice data. The input is the voice of the role-playing, and the output is the recorded voice data.

[1588] Step 2:

[1589] The terminal uploads the recorded voice data to the server. The user performs this operation by selecting the voice file through a dedicated application or a web interface and pressing the "upload" button. The input is the recorded voice data, and the output is the voice data uploaded to the server.

[1590] Step 3:

[1591] The server receives the uploaded voice data and transcribes it using a speech recognition service such as Google Cloud Speech-to-Text API. The server sends the voice data to the API and stores the returned text data. The input is the uploaded voice data, and the output is the transcribed text data.

[1592] Step 4:

[1593] The server inputs the transcribed text data into a generative AI model (e.g., OpenAI's GPT-3). The server sends the text data to the generative AI model as a prompt sentence to generate feedback content. The input is the transcribed text data, and the output is the generated feedback content.

[1594] Step 5:

[1595] The server uses a sentiment analysis tool such as IBM Watson Tone Analyzer to analyze user sentiment from the transcribed text data. The server sends the text data to the sentiment analysis tool and stores the returned sentiment analysis results. The input is the transcribed text data, and the output is the sentiment analysis results.

[1596] Step 6:

[1597] The server integrates the feedback content obtained from the generative AI model with the sentiment analysis results. The server reflects the sentiment analysis results in the feedback content to generate the final feedback. The input is the generated feedback content and the sentiment analysis results, and the output is the integrated final feedback.

[1598] Step 7:

[1599] The server provides the final feedback to the user, who can review the feedback through a dedicated application or a web interface. The input is the integrated final feedback, and the output is the feedback provided to the user.

[1600] (Application example 1)

[1601] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1602] In the conventional training system for new crew members, the content of role-playing was recorded and transcribed, making it difficult to analyze emotions and improve the quality of feedback. In addition, the feedback was one-sided and it was not possible to provide specific advice that took emotional reactions into account. This made it difficult to effectively support the growth of new crew members.

[1603] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for recording the role-playing content of the new employee, a means for transcribing the recorded content, a means for inputting the transcribed content into the generative AI model, a means for systematizing the feedback content as a professional role in sales training, a means for analyzing the user's emotions using an emotion engine, a means for reflecting the results of the emotion analysis in the feedback content, and a means for simultaneously training the sales representative and the crew. This makes it possible to provide specific and useful feedback that takes into account emotional reactions.

[1604] A "newcomer" refers to an employee who has been newly hired by a company or organization.

[1605] "Role playing" refers to a training method in which actual tasks or situations are simulated by playing specific roles.

[1606] "Recording" refers to the act of saving sound on a recording medium.

[1607] "Transcription" refers to the process of converting audio data into text data.

[1608] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or data.

[1609] "Feedback" refers to the act of providing evaluation and suggestions for improvement for specific actions or results.

[1610] An "emotion engine" refers to a system that analyzes emotions from voice and text and outputs the results.

[1611] "Sales training" refers to the training and education provided to sales representatives to acquire the necessary knowledge and skills.

[1612] A "professional role" refers to a role played by someone with specialized knowledge or experience.

[1613] "Systematization" refers to organizing and structuring information and data.

[1614] "Development" refers to education and training to improve specific skills and abilities.

[1615] A system for carrying out this invention has the following configuration. First, a user performs role-playing using a smartphone. The smartphone's microphone is used to record the voice of the role-playing. The recorded voice is saved using a recording application in the smartphone (e.g., Android (registered trademark)'s MediaRecorder class).

[1616] Next, the recorded voice file is transcribed using a speech recognition API (e.g., Google Cloud Speech-to-Text API). The transcribed text is input to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions. The emotion engine estimates emotions from the tone of voice, speaking style, and choice of words, and outputs the results.

[1617] The transcribed text and the results of the sentiment analysis are then input into a generative AI model (e.g., OpenAI GPT-3), which uses this information to generate feedback that is then displayed in the smartphone application UI and provided to the user.

[1618] As a concrete example, by inputting the following prompt sentence into the generative AI model, feedback reflecting the results of sentiment analysis can be obtained.

[1619] Prompt statement:

[1620] Below is the text of a role-playing session for a new crew member. Please provide your feedback on this, taking into account the results of the sentiment analysis.

[1621] Role-playing content:

[1622] "Hello, welcome. What item would you like to purchase today?"

[1623] Sentiment analysis results:

[1624] Tone: Friendly

[1625] Emotion: Joy

[1626] feedback:

[1627] By inputting this prompt sentence into the generative AI model, feedback such as "Your greeting was very friendly and left a good impression on the customer. Next time, it would be even better if you added a more specific question." In this way, new crew members can effectively train in physical stores.

[1628] The system makes it possible to provide specific, useful feedback that takes emotional responses into account, effectively supporting the development of new crew members.

[1629] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1630] Step 1:

[1631] The user uses a smartphone to perform role-playing. The smartphone's microphone is used to record the audio of the role-playing. A recording application (e.g. Android (registered trademark) MediaRecorder class) is launched, and recording begins when the user presses the record button. The recorded audio is saved in the smartphone as an audio file.

[1632] Input: User's voice

[1633] Output: Recorded audio file

[1634] Step 2:

[1635] The device sends the recorded audio file to a speech recognition API (e.g., Google Cloud Speech-to-Text API). The speech recognition API converts the audio data into text data and returns the transcribed text. The device receives this text data.

[1636] Input: Recorded audio file

[1637] Output: Transcribed text data

[1638] Step 3:

[1639] The terminal inputs the transcribed text data into an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the user's emotion from the text data and returns the emotion analysis result. The terminal receives this emotion analysis result.

[1640] Input: Transcribed text data

[1641] Output: Sentiment analysis results

[1642] Step 4:

[1643] The device inputs the transcribed text data and the sentiment analysis results into a generative AI model (e.g., OpenAI GPT-3). The generative AI model generates feedback based on this information. The device receives this feedback.

[1644] Input: Transcribed text data, sentiment analysis results

[1645] Output: The generated feedback

[1646] Step 5:

[1647] The terminal displays the generated feedback on the application UI, and the user can check the feedback through the smartphone screen.

[1648] Input: Generated feedback

[1649] Output: Feedback displayed in the application UI

[1650] In this way, users can receive specific, useful feedback that takes into account their emotional reactions to the role-playing content.

[1651] Example 2

[1652] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1653] With traditional training methods for new crew members, it was difficult to provide optimal training based on individual emotional responses and sales results. In addition, the process of converting voice data into text information and analyzing emotions was often done manually and was inefficient. This limited the effectiveness of the training program and prevented crew members from maximizing their growth.

[1654] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1655] In this invention, the server includes a means for acquiring voice data, a means for transmitting the acquired voice data to the server, a means for converting the voice data into text information, a means for sentiment analysis of the text information, and a means for optimizing the training program based on the result of the sentiment analysis. This makes it possible to efficiently perform a series of steps from automatic acquisition of voice data, conversion to text information, sentiment analysis, and optimization of the training program.

[1656] "Audio data" refers to data in which audio is recorded in digital format.

[1657] A "server" is a computer system that processes and stores data over a network.

[1658] "Text information" refers to voice data converted into text format.

[1659] "Sentiment analysis" is the process of analyzing text data for emotions and emotional trends.

[1660] A "development program" is a training plan designed to improve the skills and knowledge of new crew members.

[1661] "Means for acquiring" refers to a device or method for collecting voice data.

[1662] The "transmitting means" refers to a device or method for transferring the acquired voice data to the server.

[1663] The "converting means" refers to a device or method for converting voice data into text information.

[1664] An "optimizing means" is a device or method for adjusting a training program based on the results of sentiment analysis.

[1665] The present invention is a system for acquiring voice data, converting it into text information, performing sentiment analysis, and optimizing a training program based on the results. A specific embodiment of this system will be described below.

[1666] First, the user uses a device such as a smartphone or tablet to record a conversation with a customer. The user starts the recording app on the device and presses the "Start Recording" button to record the conversation. When the conversation is over, the user presses the "Stop Recording" button to end the recording.

[1667] Next, the device transmits the recorded voice data to the server. The device selects the recording file and presses the "send" button to upload the voice data to the server via the Internet.

[1668] The server converts the received voice data into text using a voice recognition service such as Google's voice recognition API or IBM's Watson. The server sends the voice data to the voice recognition API, which analyzes the voice data and generates text information. The generated text information is then sent back to the server.

[1669] The server then inputs the text information into the emotion engine to perform emotion analysis. The server sends the text information to the emotion engine, which analyzes the text information and generates emotion analysis results. The results are sent back to the server.

[1670] Finally, the server optimizes the training program for new crew members based on the results of the sentiment analysis. The server evaluates the results of the sentiment analysis and analyzes how new crew members emotionally react to specific scenarios. For example, if it is analyzed that a new crew member feels anxious when asked a tough question by a customer, the server generates a training program to strengthen the ability to respond to that scenario.

[1671] As a concrete example, a user uses a smartphone to record a conversation with a customer, converts the voice data into text, and then performs sentiment analysis on the text. The following is an example of a prompt sentence that optimizes the training program for new crew members based on the results.

[1672] Example prompt:

[1673] "Record conversations with customers and convert the audio data into text. Then run the text through sentiment analysis and use the results to optimize your onboarding program."

[1674] In this way, the system can efficiently perform a series of processes from acquiring voice data to analyzing emotions and optimizing training programs.

[1675] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1676] Step 1:

[1677] Acquiring audio data

[1678] The user uses a device such as a smartphone or tablet to record a conversation with a customer. The user starts the recording app on the device and presses the "Start Recording" button to record the conversation. When the conversation is over, the user presses the "Stop Recording" button to end the recording.

[1679] Input: Customer conversation

[1680] Output: Recorded audio data

[1681] Step 2:

[1682] Sending voice data to the server

[1683] The device sends the recorded voice data to the server. The device selects the recording file and presses the "Send" button to upload the voice data to the server via the Internet.

[1684] Input: Recorded audio data

[1685] Output: Audio data uploaded to the server

[1686] Step 3:

[1687] Converting voice data into text

[1688] The server converts the received voice data into text using a voice recognition service such as Google's voice recognition API or IBM's Watson. The server sends the voice data to the voice recognition API, which analyzes the voice data and generates text information. The generated text information is then sent back to the server.

[1689] Input: Audio data uploaded to the server

[1690] Output: Text information

[1691] Step 4:

[1692] Conducting sentiment analysis

[1693] The server inputs the text information into the emotion engine and performs sentiment analysis. The server sends the text information to the emotion engine, which analyzes the text information and generates the result of the sentiment analysis. The result is sent back to the server.

[1694] Input: Text information

[1695] Output: Sentiment analysis results

[1696] Step 5:

[1697] Optimizing training programs

[1698] The server optimizes the training program for new crew members based on the results of the sentiment analysis. The server evaluates the results of the sentiment analysis and analyzes how new crew members react emotionally to specific scenarios. For example, if it is analyzed that a new crew member feels anxious when asked a tough question by a customer, a training program is generated to strengthen the ability to respond to that scenario.

[1699] Input: Sentiment analysis results

[1700] Output: Optimized breeding program

[1701] (Application example 2)

[1702] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1703] In conventional sales training systems, it was difficult to grasp the emotional state of new crew members in real time and provide appropriate feedback based on that. Therefore, when new crew members felt anxious or stressed, it was not possible to respond appropriately to their emotions, which led to a problem of reduced training effectiveness.

[1704] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1705] In this invention, the server includes a means for recording the role-playing content of the new crew member, a means for transcribing the recorded content, a means for inputting the transcribed content into a sentiment analysis engine, a means for providing feedback based on the result of the sentiment analysis, and a means for simultaneously training the sales representative and the crew member. This makes it possible to grasp the emotional state of the new crew member in real time and provide appropriate feedback based on that.

[1706] The "means for recording the role-playing content of a new crew member" refers to a device or method for acquiring audio data of the role-playing performed by a new crew member.

[1707] A "means for transcribing recorded content" is a device or method for converting acquired voice data into text data.

[1708] "Means for inputting the transcribe...

Claims

1. A means for acquiring voice data of the newcomer's role-playing; A means for converting the acquired voice data into text information; a means for generating a prompt sentence for instructing an evaluation for the evaluation item based on the character information; A means for generating feedback using the prompt sentence and a generative AI model; A system including:

2. a means for performing emotion analysis using an emotion engine to analyze the emotion of the new employee from one or both of the voice data and the character information, The prompt sentence is generated based on the character information and a result of the sentiment analysis. The system of claim 1 .

3. The evaluation items include at least one of clarity of pronunciation, flow of speech, naturalness of dialogue, and overall evaluation. A system according to claim 1 or claim 2.

Images