System

Abstract

A system is provided.SOLUTION: A system, comprising: means for a user to reserve a session with an expert; means for sending reservation information of the user to a server, wherein the server stores the reservation information in a database; and means for providing a generated reservation ID to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many learners seek individualized instruction and professional advice from experts to improve their skills, but booking sessions with experts and receiving feedback is often cumbersome and time-consuming. Additionally, converting audio data into text and text into audio during the expert instruction process is also labor-intensive. These barriers hinder learners' skill improvement. [Means for solving the problem]

[0005] To solve this problem, we provide a system that allows users to easily reserve sessions with experts. Specifically, we build a system that includes a means for users to reserve sessions with experts, a means for sending the user's reservation information to a server and for the server to store the reservation information in a database, and a means for providing the user with a generated reservation ID. Furthermore, we provide a system that includes a means for sending the user's voice data to the server and for the server to convert the voice data to text, a means for inputting the text data to a generation artificial intelligence (AI) that generates a response, and a means for converting the generated response text into voice and providing it to the user. We also provide a system that allows learners to smoothly receive expert guidance and effectively improve their skills by including a means for receiving a session completion notification from the user, analyzing the session content, and creating a feedback report, and a means for sending the feedback report to the user.

[0006] "User" means an individual or entity that uses the System to book and attend sessions with an Expert.

[0007] An "expert" is someone who has specialized knowledge and experience in a particular field and provides guidance and advice to learners.

[0008] A "session" refers to an activity involving online or offline interaction or instruction between a user and an expert at a specific date and time.

[0009] "Reservation information" refers to data necessary to conduct a session (user ID, expert ID, date and time, etc.).

[0010] A "server" refers to a computer system that receives data sent by users, processes it, and stores it in a database.

[0011] "Database" refers to a system for storing and managing data such as reservation information and session content.

[0012] "Reservation ID" refers to a unique identifier provided to a user when a reservation is confirmed.

[0013] "Audio data" refers to a digital audio file that records what a user has said.

[0014] "Text data" refers to voice data converted into text information, or the text information of a response generated by a generation AI.

[0015] "Generative AI" refers to AI techniques that generate appropriate responses based on user input.

[0016] "Feedback Report" refers to a document containing analytical results that evaluates the content and progress of a session and is provided to the user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] Based on the problems to be solved by the present invention, specific embodiments for carrying out the invention will be described below.

[0039] 1. Reservation System Implementation

[0040] The user opens an interface on the terminal to reserve a session with an expert. In this interface, the user enters their ID, the desired expert, and the date and time of the session. The terminal sends this reservation information to the server. The server receives the reservation information and stores it in a database. The server then generates a unique reservation ID and returns it to the user.

[0041] 2. Session Format

[0042] On the day of the session, the user asks the expert a question by voice in order to participate in the session. The terminal records the user's voice and sends the voice data to the server. The server analyzes the received voice data and converts it into text using voice recognition technology. The converted text data is input into a generative AI system. The generative AI system generates an appropriate response to the user's question. This response text is again converted into voice by the server and sent to the user's terminal. The terminal plays back the voice data and provides the response to the user.

[0043] 3. Feedback System Embodiments

[0044] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress. It creates a feedback report based on the analysis results and sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[0045] Specific examples

[0046] Consider a specific example where a user wants to book a session with an expert. If User A wants to book a session with Expert B at 3:00 PM on November 1st, User A enters his / her information and desired date and time into the booking interface. The terminal sends this to the server, which stores the booking information in a database, generates a booking ID 12345, and notifies User A.

[0047] On the day of the session, User A voice-inputs the question, "What should I do to learn a new programming language?" The device records the voice and sends it to the server. The server converts the voice data into text and inputs it into the generative AI. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with practical projects." This text response is converted back into speech and provided to User A.

[0048] When the session ends, User A sends a completion notification, and the server analyzes the session content and creates a feedback report. The feedback report includes an evaluation such as "User A understands basic knowledge, but more practical projects would further improve his skills." The user can use this feedback to plan his next learning experience.

[0049] In this way, a system can be implemented that allows users to efficiently improve their skills while receiving guidance from experts.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] The user opens an interface on the terminal to reserve a session with an expert, and enters the necessary reservation information, such as the user's ID, the desired expert, and the desired session date and time.

[0053] Step 2:

[0054] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[0055] Step 3:

[0056] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[0057] Step 4:

[0058] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[0059] Step 5:

[0060] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[0061] Step 6:

[0062] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[0063] Step 7:

[0064] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[0065] Step 8:

[0066] The server analyzes the received voice data and converts it into text using speech recognition, which is then input into the conversational AI system.

[0067] Step 9:

[0068] The server uses a conversational AI system to generate appropriate responses to user questions and challenges, and the generated text responses are stored.

[0069] Step 10:

[0070] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[0071] Step 11:

[0072] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[0073] Step 12:

[0074] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[0075] Step 13:

[0076] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio and text data collected during the session.

[0077] Step 14:

[0078] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[0079] Step 15:

[0080] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[0081] Step 16:

[0082] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[0083] Step 17:

[0084] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional systems for booking meetings with experts require a complicated process from reservation to actual meeting, and lack the technology to make it more efficient. Furthermore, evaluating users' learning progress and providing feedback after the meeting is time-consuming, which reduces the overall quality of the service. Another problem is the lack of an established process for efficiently converting voice input into text and generating appropriate responses using artificial intelligence.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes: means for a user to reserve a meeting with an expert; means for transmitting the user's reservation information to a communication device and for the communication device to store the reservation information in a storage device; means for providing the user with a generated reservation identifier; means for the communication device to convert voice data into text; means for inputting the generated text data into an artificial intelligence system and for the artificial intelligence system to generate a response; and means for converting the generated response text into voice and providing it to the user. This enables a consistent and efficient process from reserving a meeting to holding it and providing feedback after it has ended.

[0090] "Users" are those who use the system to schedule meetings with experts and ask questions or provide feedback.

[0091] An "expert" is a person from whom a user can meet and receive knowledge and advice.

[0092] A "meeting" is a session in which users and experts interact and ask questions.

[0093] "Reservation information" refers to information that a user enters for a meeting, such as their ID, the desired expert, and the date and time of the meeting.

[0094] A "communication device" is an electronic device that transmits user input data and voice data to a server and conveys the user's response.

[0095] A "storage device" is a storage system that includes a database in which user reservation information and the like are stored.

[0096] A "Reservation Identifier" is a generated unique ID that identifies a particular reservation.

[0097] "Voice data" is digital data of the voice input by the user to ask a question to an expert.

[0098] "Text data" is character information converted from voice data using voice recognition technology.

[0099] An "artificial intelligence system" is a machine learning-based system that generates appropriate responses based on a user's text data.

[0100] "Response text" is a textual response to a user's question generated by an artificial intelligence system.

[0101] "Speech synthesis technology" is a technology for converting text data into voice data.

[0102] An "evaluation report" is a document summarizing the results of analyzing the session content and evaluating the user's learning progress.

[0103] MODE FOR CARRYING OUT THE INVENTION

[0104] The system of the present invention is designed to efficiently and automatically carry out the entire process of a user reserving a meeting with an expert, conducting the meeting, and receiving feedback after the meeting. This system operates mainly through three parties: a server, a terminal, and a user.

[0105] Reservation System Embodiment

[0106] Users can book a meeting with an expert using a dedicated interface. Specifically, users enter their ID, the desired expert, and the date and time of the meeting. This interface is built using libraries such as React and Angular on the front end, Node.js and Django on the back end, and MySQL and PostgreSQL as the database.

[0107] The terminal sends the collected reservation information to the server, which stores the received reservation information in a database, generates a unique reservation identifier (reservation ID), and returns it to the user.

[0108] For example, the user enters "userA, expert B, 2023-11-01 15:00" and the terminal sends the reservation information to the server. The server saves the reservation information in a storage device, generates a reservation ID "12345", and returns it to the user.

[0109] Meeting System Embodiments

[0110] On the day of the meeting, the user speaks their question to the expert. The device records the user's voice data and sends it to the server, which then converts the voice data into text using a speech recognition service (such as Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[0111] The converted text data is then fed into a generative AI model (such as OpenAI GPT-4 or Google BERT) to generate an appropriate response. The generated response text is then converted into audio data using speech synthesis technology and sent to the user's device. The device then plays back the audio data and provides it to the user.

[0112] For example, a user may ask, "What should I do to learn a new programming language?" The server converts the voice data into text, and the generative AI generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This response is then converted into audio and provided to the user.

[0113] Feedback System Embodiments

[0114] After the meeting, the user sends a notification to the system. The device then sends this notification to the server, which then analyzes the session content and evaluates the user's learning progress. Machine learning techniques can be used for the analysis.

[0115] The server then creates a feedback report based on the evaluation results and sends it to the user, who can review the feedback report and use it to plan their next study.

[0116] As a concrete example, a user sends a completion notification to the server, and the server generates a feedback report stating, "User A understands basic knowledge, but his skills will improve further with more practical projects," and sends it to the user.

[0117] Prompt Sentence Examples

[0118] Please provide a detailed explanation, using examples, of how a user schedules a meeting with an expert, conducts the meeting, and receives feedback. Please also include specific actions taken by the server, device, and user during this process.

[0119] This system allows users to efficiently go through a series of processes while receiving expert guidance.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Specific explanation of processing steps

[0122] Reservation System

[0123] Step 1: View the interface

[0124] The terminal displays an interface for the user to schedule a meeting with an expert, the interface including fields for entering the user ID, the desired expert, and the date and time of the meeting.

[0125] input

[0126] User operation (interface display request)

[0127] output

[0128] View the booking interface

[0129] Specific actions

[0130] The terminal loads the reservation interface HTML page and displays it to the user.

[0131] Step 2: Enter reservation information

[0132] The user enters their ID, the desired expert, and the date and time of the meeting into the interface.

[0133] input

[0134] User input information (ID, expert, date and time)

[0135] output

[0136] Reservation information entered

[0137] Specific actions

[0138] The user enters information using a keyboard or touch screen.

[0139] Step 3: Submit your reservation information

[0140] The terminal transmits the reservation information input by the user to the server.

[0141] input

[0142] Reservation information (ID, specialist, date and time)

[0143] output

[0144] Sending data to the server

[0145] Specific actions

[0146] The terminal sends the reservation information to the server as an HTTP request.

[0147] Step 4: Save your reservation

[0148] The server stores the received reservation information in a database.

[0149] input

[0150] Received reservation information

[0151] output

[0152] Reservation information recorded in the database

[0153] Specific actions

[0154] The server converts the reservation information into the appropriate format and stores it in a database using SQL.

[0155] Step 5: Generate a reservation ID

[0156] The server generates a unique reservation identifier (reservation ID).

[0157] input

[0158] Saved reservations

[0159] output

[0160] Generated reservation ID

[0161] Specific actions

[0162] The server runs an algorithm to generate a unique ID based on the reservation information.

[0163] Step 6: Reservation ID response

[0164] The server returns the generated reservation ID to the user.

[0165] input

[0166] Generated reservation ID

[0167] output

[0168] Sending the reservation ID to the user

[0169] Specific actions

[0170] The server creates an HTTP response including the reservation ID and sends it to the terminal.

[0171] Meeting System

[0172] Step 1: Enter your voice question

[0173] On the day of the session, the user will use the terminal to voice-type questions to the expert.

[0174] input

[0175] User voice questions

[0176] output

[0177] Audio data

[0178] Specific actions

[0179] The user speaks a question into the microphone.

[0180] Step 2: Sending audio data

[0181] The terminal records the user's voice and transmits the voice data to the server.

[0182] input

[0183] Audio data

[0184] output

[0185] Sending audio data to the server

[0186] Specific actions

[0187] The device records the audio and sends it to the server in WAV or MP3 format.

[0188] Step 3: Convert audio data to text

[0189] The server converts the received voice data into text using voice recognition technology.

[0190] input

[0191] Audio data

[0192] output

[0193] Text data

[0194] Specific actions

[0195] The server calls a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the speech into text.

[0196] Step 4: AI input of text data

[0197] The server inputs the converted text data into a generative AI model.

[0198] input

[0199] Text data

[0200] output

[0201] Input data for AI models

[0202] Specific actions

[0203] The server converts the text data into a format suitable for the AI ​​model and sends it to the model.

[0204] Step 5: Generate an AI response

[0205] Generative AI models (e.g., OpenAI GPT-4) generate appropriate responses to user questions.

[0206] input

[0207] Input data for AI models

[0208] output

[0209] Response Text

[0210] Specific actions

[0211] The AI ​​model performs the calculations to generate an answer to the question.

[0212] Step 6: Convert response text to speech

[0213] The server converts the generated response text into voice data using voice synthesis technology.

[0214] input

[0215] Response Text

[0216] output

[0217] Audio data

[0218] Specific actions

[0219] The server calls a speech synthesis API (e.g., Amazon Polly) to convert the text to speech.

[0220] Step 7: Sending a response audio

[0221] The server transmits the generated voice data to the user's terminal.

[0222] input

[0223] Audio data

[0224] output

[0225] Sending audio to the user

[0226] Specific actions

[0227] The server sends the audio data to the terminal as an HTTP response.

[0228] Step 8: Play the audio response

[0229] The terminal plays this audio data and provides a response to the user.

[0230] input

[0231] Audio data

[0232] output

[0233] Playing audio

[0234] Specific actions

[0235] The device plays the audio data using a playback application.

[0236] Feedback System

[0237] Step 1: Sending a termination notice

[0238] After the meeting is over, the user sends a notification of the end to the system.

[0239] input

[0240] Termination Notice

[0241] output

[0242] Sending notifications to the server

[0243] Specific actions

[0244] The user clicks the end meeting button and a notification is sent to the server.

[0245] Step 2: Receive notifications

[0246] The terminal sends a termination notification to the server.

[0247] input

[0248] Termination Notice

[0249] output

[0250] Sending notifications to the server

[0251] Specific actions

[0252] The terminal sends a termination notification to the server as an HTTP request.

[0253] Step 3: Analyzing session content

[0254] The server analyzes the session content based on the received notification and evaluates the user's learning progress.

[0255] input

[0256] End notice and session details

[0257] output

[0258] Evaluation Data

[0259] Specific actions

[0260] The server analyzes the session data using machine learning algorithms.

[0261] Step 4: Create a feedback report

[0262] The server generates a feedback report based on the analysis results.

[0263] input

[0264] Evaluation Data

[0265] output

[0266] Feedback Report

[0267] Specific actions

[0268] The server runs an algorithm that automatically generates a feedback report.

[0269] Step 5: Submit the report

[0270] The server sends the generated feedback report to the user.

[0271] input

[0272] Feedback Report

[0273] output

[0274] Sending reports to users

[0275] Specific actions

[0276] The server sends the feedback report as an HTTP response.

[0277] Step 6: Review the report

[0278] Users can check the feedback report and use it to plan their future studies.

[0279] input

[0280] Feedback Report

[0281] output

[0282] Verified Feedback

[0283] Specific actions

[0284] The user displays the feedback report on the device and checks the contents.

[0285] (Application example 1)

[0286] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0287] Current factory operations often rely on human resources for troubleshooting and work guidance, which can lead to reduced productivity and the need for rapid response when problems occur. Furthermore, it can take time for staff to learn new technologies and operating procedures, making it difficult for them to respond immediately. Therefore, there is a need for a system that can provide real-time problem solving and guidance through a voice interface.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0289] In this invention, the server includes a means for recognizing user voice data and converting it into text data, a means for inputting the text data into a generating artificial intelligence, which then generates a response, and a means for providing the generated voice response to the user, thereby enabling quick troubleshooting and work guidance within the factory.

[0290] "User" refers to the person or factory staff who uses the system.

[0291] An "expert" is a person or system with advanced specialized knowledge or skills.

[0292] A "session" is a period of time during which information is exchanged or consulted between a user and an expert.

[0293] A "reservation" is a prior application or registration to use a system or expert at a specific date and time.

[0294] A "server" is a computer system that processes reservation information and voice data and inputs them into the generation AI.

[0295] "Voice data" refers to data in which a user's questions or requests are recorded in the form of sound.

[0296] "Text data" refers to data obtained by converting voice data into character information.

[0297] "Generative artificial intelligence" refers to an AI system that generates appropriate responses based on input text data.

[0298] "Response text" refers to data that expresses the answer content generated by the generating AI in text.

[0299] A "feedback report" is a report that summarizes the user's learning progress and areas for improvement after a session.

[0300] System program description

[0301] The system for realizing this invention is constructed using the following hardware and software: a user's terminal, a server, a generative AI model, speech recognition technology, and speech synthesis technology.

[0302] Hardware and Software Use

[0303] 1. On the user's device:

[0304] Microphone (audio input)

[0305] Speaker (audio output)

[0306] Internet connection function (data communication)

[0307] 2. Server:

[0308] Database (storing reservation information)

[0309] Speech recognition technology (converts voice to text)

[0310] Generative AI models (generate responses from text)

[0311] Speech synthesis technology (converting text to speech)

[0312] Details of data processing and calculation

[0313] The server receives the user's voice data and converts it into text using speech recognition technology, a process that uses software such as speech_recognition.

[0314] The converted text data is then fed into a generative artificial intelligence system, which uses a generative AI model such as GPT-2 to generate an optimal response based on the input text.

[0315] The generated text response is converted into voice data using speech synthesis technology, using software such as pyttsx3.

[0316] Finally, this voice data is sent to the user's terminal and presented to the user through a speaker.

[0317] Specific examples

[0318] Reservation System:

[0319] Example: When a user books a session with an expert, they enter their information and desired date and time in the terminal interface. The information is sent to the server and stored in a database. The server generates a unique booking ID and sends it back to the user.

[0320] session:

[0321] Example: On the day of the session, the user voice-inputs the question, "What should I do to learn a new programming language?" The server converts the voice data into text and feeds it into a generative AI model. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This text response is converted into speech and provided to the user's device.

[0322] Prompt Sentence Examples

[0323] Here is an example prompt for the example above:

[0324] Generate an appropriate response to "What caused this machine in the factory to stop working?"

[0325] This will enable quick troubleshooting and work guidance within the factory, and is expected to improve productivity.

[0326] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0327] Step 1:

[0328] The user opens the reservation interface on the terminal to reserve a session with an expert. The user enters their information (name and ID), the desired expert, and the date and time of the session. The terminal sends the reservation information entered by the user to the server. The server receives the reservation information and stores it in a database. The server generates a unique reservation ID and returns it to the user via the terminal.

[0329] Step 2:

[0330] On the day of the session, the user uses the microphone on the device to ask the expert a question. The device records the user's voice data and sends it to the server. The input at this time is the user's voice data.

[0331] Step 3:

[0332] The server converts the received voice data into text data using speech recognition software. This process uses the speech_recognition software. By converting the voice data into text, the content of the user's question is output as text data.

[0333] Step 4:

[0334] The server inputs the user's question, converted into text, into a generative AI model. GPT-2 is used as the generative AI model. The text data is passed as input to the AI ​​model, which generates an appropriate response text. The generative AI model generates a response using a prompt sentence based on the user's question. Specifically, a response based on the prompt sentence, "Please generate an appropriate reply to the question 'What caused this machine in the factory to stop?'" is output.

[0335] Step 5:

[0336] The generated response text is converted into voice data by the speech synthesis system in the server. For this purpose, we use the pyttsx3 software, which generates voice data from the text data and outputs the response voice.

[0337] Step 6:

[0338] The server transmits the generated voice data to the terminal, which receives the voice data and provides a response to the user through a speaker, where the output is a voice response that the user can hear.

[0339] Step 7:

[0340] After the session ends, the user sends a notification to the server via their device. The server receives this notification, analyzes the session, and creates a feedback report. This analysis and feedback includes usage data, user questions, and AI responses.

[0341] Step 8:

[0342] The server sends the created feedback report to the terminal, which displays the report to the user, allowing the user to use it for future learning and work improvement. The output at this time is the feedback report.

[0343] The above process realizes a system that allows users to quickly receive expert guidance and troubleshooting suggestions in real time using voice.

[0344] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0345] Based on the problem to be solved by the present invention, a specific embodiment for carrying out the invention in combination with an emotion engine will be described below.

[0346] 1. Reservation System Implementation

[0347] The user opens an interface on their terminal to reserve a session with an expert and enters the necessary reservation information, such as their ID, desired expert, and desired session date and time. The terminal sends this reservation information to the server's reservation API via a POST request. The server receives the reservation information and generates a unique reservation ID when saving it in the database. The server then notifies the user of the generated reservation ID.

[0348] 2. Session Format

[0349] At the start of a session, the user inputs a question or task by voice. The device records the user's voice and sends the voice data to the server. The server analyzes the received voice data, converts it into text using voice recognition technology, and then recognizes the user's emotions using an emotion engine. Based on the recognized emotions, the dialogue AI system generates an appropriate response. The generated text response is converted back into voice by the server and sent to the device. The device plays back this voice data and provides the response to the user.

[0350] 3. Feedback System Embodiments

[0351] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[0352] Specific examples using emotion engines

[0353] Consider a scenario where, during a session, the user asks a question by voice, such as, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes the text data to the emotion engine. The emotion engine analyzes the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, the response generated might be, "Start with a basic tutorial, then hone your skills with practical projects." The server then converts this text back into audio data and provides it to the user.

[0354] Dynamic Adjustment Examples

[0355] The server evaluates the user's emotions in real time during the session and dynamically adjusts the expert's instruction. For example, if the user shows signs of anxiety, the expert will flexibly change the instruction to suit the situation. This allows the user to learn in a relaxed state and effectively acquire skills.

[0356] Example of a feedback report

[0357] After the session ends, the server receives a notification and analyzes the session content. This analysis includes the user's voice data, text data, and emotional history. Based on the analysis results, the server provides feedback such as, "The user has grasped basic knowledge, and in the future, more practical projects will further improve their skills." The feedback report is sent to the user, who can use it to plan their next learning.

[0358] In this way, a system can be implemented in which users can smoothly progress through learning by utilizing the emotion engine while receiving guidance from an expert, and can efficiently improve their skills.

[0359] The processing flow will be explained below.

[0360] Step 1:

[0361] The user opens an interface on the terminal for reserving a session with an expert and inputs necessary reservation information such as the user's ID, the desired expert, and the desired date and time of the session.

[0362] Step 2:

[0363] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[0364] Step 3:

[0365] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[0366] Step 4:

[0367] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[0368] Step 5:

[0369] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[0370] Step 6:

[0371] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[0372] Step 7:

[0373] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[0374] Step 8:

[0375] The server analyzes the received voice data and converts the voice data into text using speech recognition, which is then input to the emotion engine.

[0376] Step 9:

[0377] The server uses an emotion engine to recognize the user's emotions from their voice, and classifies the emotions into categories such as "happiness," "sadness," "anger," and "surprise."

[0378] Step 10:

[0379] The server inputs text data based on the user's emotions into the conversational AI system, which then generates an appropriate response. For example, if the user expresses "anxiety," the AI ​​will generate a more friendly and reassuring response.

[0380] Step 11:

[0381] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[0382] Step 12:

[0383] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[0384] Step 13:

[0385] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[0386] Step 14:

[0387] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio data, text data, and emotion history collected during the session.

[0388] Step 15:

[0389] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[0390] Step 16:

[0391] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[0392] Step 17:

[0393] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[0394] Step 18:

[0395] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[0396] Example 2

[0397] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Conventional reservation systems have the drawback of requiring users to perform many manual operations when reserving a session with an expert, and making it difficult to manage reservation information. Furthermore, they are unable to grasp the user's emotions in real time during the session and dynamically adjust the expert's instruction content based on those emotions, resulting in insufficient learning effectiveness. Furthermore, feedback after the session is sometimes delayed, making it difficult to adequately support the user's learning progress.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0400] In this invention, the server includes means for transmitting the user's reservation information to the server and saving the reservation information in a database, means for generating a specific reservation number and providing it to the user, means for transmitting voice data from the user to the server, means for converting the voice data to text, means for inputting the text data into an emotion analysis engine and recognizing emotions, means for generating a response using a generative AI model based on the recognized emotions, means for converting the generated response text into voice and providing it to the user, means for receiving a session end notification from the user and analyzing the session content and the user's emotion history to make a final evaluation, and means for creating a feedback report and sending it to the user. This enables the user to make a reservation smoothly, receive appropriate responses in real time during the session, and receive prompt feedback after the session ends.

[0401] A "user" is a person who utilizes the system to book and conduct sessions with experts.

[0402] An "expert" is someone who has extensive knowledge and experience in a particular field and who can provide guidance and advice to users.

[0403] A "server" is a computer system that receives information from users, processes the data, stores it in a database, and runs generative AI models.

[0404] A "terminal" is a device used by a user to access the system, and includes a PC, smartphone, tablet, etc.

[0405] A "generative AI model" is an artificial intelligence algorithm that generates optimal responses or predictions based on specific input data.

[0406] A "reservation number" is a unique identifier that allows a user to identify a reservation.

[0407] "Voice data" refers to data in which the user's voice is recorded in digital format.

[0408] "Text data" refers to character data obtained by analyzing and transcribing voice data.

[0409] An "emotion analysis engine" is an analysis device or program that analyzes a user's emotions (e.g., excitement, anxiety, joy, etc.) from input text data.

[0410] A "feedback report" is a report that includes an evaluation and suggestions for future learning, generated based on the user's performance and emotional history during the session.

[0411] This invention is a system that allows users to reserve sessions with experts, and receive feedback on the sessions. This system is composed of three entities: a server, a terminal, and a user, and operates as follows.

[0412] 1. Reservation System Implementation

[0413] To book a session with an expert, a user opens the device's interface and enters the necessary information, such as their ID, the preferred expert, and the desired date and time of the session. This interface is provided as a web page or mobile app. The device sends this reservation information to the server's reservation API via an HTTP POST request. The server executes an SQL query to store the received reservation information in a database and generates a unique reservation number. The server then returns a response including the generated reservation number to the device, and the user is notified. This allows the user to confirm that the reservation was successful.

[0414] 2. Session Format

[0415] At the start of a session, the user uses the voice input function to speak a question or task into the device. The device records the user's voice and sends the audio data to the server. This audio data is saved in an appropriate format (such as .wav or .mp3). The server analyzes the received audio data and converts it into text using speech recognition technology (e.g., Google Cloud Speech-to-Text). The server then passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. Based on the recognized emotion, a generative AI model (e.g., OpenAI GPT-3) generates a response. This response is again converted into speech by the server (e.g., Amazon Polly) and sent to the device. The device plays the audio data and provides the response to the user.

[0416] 3. Feedback System Embodiments

[0417] After the session ends, the user sends a completion notification to the system. The device then sends this completion notification to the server. The server analyzes the session content based on the received completion notification (e.g., using an NLP analysis tool) and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning.

[0418] Add specific examples to the description

[0419] Reservation system example

[0420] For example, if a user wants to reserve a session with an expert "Expert A" with ID "user123" for 15:00 on November 1, 2023, he or she enters this information into the terminal interface. The terminal sends this information to the server, which stores it in its database, generates a reservation number "reserve456" and notifies the user.

[0421] Session Example

[0422] For example, if a user asks, "What should I do to learn a new programming language?", the device records the question and sends it to a server. The server converts the speech to text using Google Cloud Speech-to-Text and analyzes sentiment using IBM Watson Tone Analyzer. OpenAI GPT-3 generates a response such as "Start with basic tutorials and hone your skills with hands-on projects," and Amazon Polly converts this text to speech and serves it to the user.

[0423] Examples of feedback systems

[0424] After the session ends, the server receives a notification from the user and analyzes the user's emotional history and session data. It then generates feedback to the user, such as "The user has grasped basic knowledge and would benefit from more practical projects."

[0425] Prompt Sentence Examples

[0426] "What should I do when learning a new programming language?"

[0427] Through each of these steps, users can enjoy a smooth booking, session, and feedback experience, and a system can be implemented that allows them to efficiently improve their skills while receiving expert guidance.

[0428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0429] Step 1:

[0430] The user opens the device interface and enters information such as their ID, the preferred specialist, and the desired date and time of the session. This information is entered into the device as reservation information. The device temporarily stores this information and displays a confirmation screen to the user.

[0431] Step 2:

[0432] When the user confirms the reservation information and presses the send button, the device sends this reservation information to the server's reservation API via an HTTP POST request. The input data is the user ID, expert ID, and reservation date and time, and is sent to the server in JSON format.

[0433] Step 3:

[0434] The server analyzes the reservation information received from the terminal and generates an SQL query to store it in the database. Here, the user ID, expert ID, and reservation date and time are input data, and are stored in fields in the "Reservation" table. The server generates a unique reservation number and returns it as a response.

[0435] Step 4:

[0436] The terminal receives the response from the server and displays a confirmation message to the user, including the reservation number. This confirms that the reservation was successfully completed. The output is the reservation number and the confirmation message.

[0437] Step 5:

[0438] At the scheduled time, the user starts the session using the device. Using the voice input function, the user voice-enters questions and tasks into the device. This voice data becomes the input data.

[0439] Step 6:

[0440] The device records the user's voice and sends it to the server. The voice data is encoded in the appropriate format (e.g., .wav or .mp3). The data sent to the server is the binary data of the voice file.

[0441] Step 7:

[0442] The server analyzes the received voice data and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The input data is an audio file, and the output is text data. The server temporarily stores the text data.

[0443] Step 8:

[0444] The server passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data, and the output is the emotion recognition result (e.g., excitement, anxiety, etc.). The server temporarily stores the emotion recognition result.

[0445] Step 9:

[0446] The server inputs the recognized emotion and text data into a generative AI model (e.g., OpenAI GPT-3) to generate an appropriate response. The input data is text and emotion recognition results, and the output is a response text. The server temporarily stores the generated response text.

[0447] Step 10:

[0448] The server converts the generated response text into speech using speech synthesis technology (e.g., Amazon Polly). The input data is the response text, and the output is an audio file. The server then sends the audio data to the device.

[0449] Step 11:

[0450] The terminal plays back the voice data received from the server and provides a response to the user. The output is a voice response, allowing the user to experience a dialogue in real time.

[0451] Step 12:

[0452] After the session ends, the user sends a termination notification to the system. By clicking the end button, the terminal sends a termination notification to the server. The data sent is a termination notification flag.

[0453] Step 13:

[0454] The server receives the termination notification and analyzes the audio data, text data, and emotion history during the session (e.g., using an NLP analysis tool). The input data are audio data, text data, and emotion recognition results, and the output is the analysis results.

[0455] Step 14:

[0456] The server generates a feedback report based on the analysis results. The input data is the analysis results, and the output is the feedback report. The report contains information that is useful for the user's learning progress and future learning plans.

[0457] Step 15:

[0458] The server sends a feedback report to the user, and the sent data is the feedback report, which the user can check and use for future learning.

[0459] (Application example 2)

[0460] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0461] Conventional online session systems lack support for users to communicate effectively with experts. Furthermore, because they respond uniformly without considering the user's emotions, it is difficult to provide appropriate support for emotions such as anxiety and excitement. Furthermore, because feedback does not take into account the user's emotions or learning progress, it is difficult to create an effective learning plan.

[0462] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing user reservation information in a database, means for analyzing the user's voice input in real time based on the generated reservation information and recognizing emotions, means for generating an appropriate response to the voice input based on the emotion recognition, and means for dynamically adjusting and reproducing the response taking the user's emotions into consideration. This makes it possible to grasp the user's emotions in real time and provide an optimal response. In addition, by evaluating the user's learning progress and providing appropriate feedback, it is possible to create an effective learning plan.

[0463] "User" refers to a person who uses the system to hold a session with an expert.

[0464] An "expert" refers to a person or system that has specialized knowledge and skills in a particular field and that holds a session with a user to provide assistance.

[0465] A "session" refers to a period of interaction or instruction between a user and an expert for a specific purpose.

[0466] "Reservation information" refers to information provided by a user when reserving a session with an expert, and includes, for example, an ID, a desired expert, and a desired date and time for the session.

[0467] "Server" refers to a central processing unit that processes data and communicates between users and experts.

[0468] "Database" refers to a system that centrally manages and stores data such as reservation information.

[0469] A "reservation ID" is a unique identifier generated by the server and used to manage session reservations.

[0470] "Analyzing in real time" refers to a process of instantly analyzing voice data while the user is inputting voice data.

[0471] "Emotion recognition" refers to analyzing and identifying emotional states such as excitement, anxiety, and joy from a user's voice and text data.

[0472] "Dynamic response adjustment" refers to changing and optimizing expert guidance and system responses in real time according to the user's emotional state.

[0473] A "feedback report" is a report generated by the server after a session ends. It evaluates the user's learning progress and emotional history, and provides information that can be used as a reference for future learning plans.

[0474] The present invention provides a product purchasing support system in a virtual store that allows users to reserve a session with an expert and dynamically responds to user emotions. Specific embodiments of this system are described below.

[0475] 1. Reservation System Implementation

[0476] A user uses their device (e.g., smartphone, tablet, or computer) to reserve a session with an expert. The user opens the reservation interface and enters the necessary information, such as their ID, the desired expert, and the desired session date and time. The device then sends this reservation information to the server's reservation API via a POST request. The server stores the received reservation information in a database and generates a unique reservation ID. This reservation ID is notified to the user.

[0477] 2. Session Format

[0478] At the start of a session, the user vocally inputs a question or task related to the session content. The device records the user's voice and sends the voice data to the server. The server then uses speech recognition technology (e.g., Google Speech Recognition API) to convert the voice data into text. It then uses an emotion recognition engine (e.g., EmotionRecognizer library) to analyze and recognize the user's emotions from the text data. Based on the recognized emotions, the server's conversational AI system (e.g., generative AI model) generates an appropriate response. The generated text response is converted into voice data by the server (e.g., TextToSpeech library), sent to the device, and provided to the user.

[0479] 3. Feedback System Embodiments

[0480] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning. In this way, the user can effectively progress with their learning by utilizing the emotion engine while receiving guidance from an expert.

[0481] Specific examples

[0482] For example, consider a case where a user asks by voice, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes this text data to an emotion recognition engine. The emotion recognition engine analyzes emotions from the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, it might generate a response as text, such as "Start with a basic tutorial and then hone your skills with practical projects," and the server then converts this text back into audio data and provides it to the user.

[0483] Prompt Sentence Examples

[0484] User Question: "What should I do when learning a new programming language?"

[0485] Recognized emotion: "excitement"

[0486] This allows users to receive expert guidance in a way that suits their emotions, resulting in a more effective and personalized purchasing experience and learning.

[0487] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0488] Step 1:

[0489] The user opens the reservation interface and enters reservation information such as ID, desired expert, desired session date and time, etc. The terminal sends this input information to the server. The server stores the received reservation information in a database and generates a unique reservation ID. The server notifies the user of this reservation ID.

[0490] Step 2:

[0491] At the start of a session, the user enters a question or task by voice through the device. The device records the user's voice and sends the voice data to the server. The server receives the voice data and converts it into text using voice recognition technology.

[0492] Step 3:

[0493] The server inputs the converted text data into an emotion recognition engine, which analyzes the text data and identifies the user's emotion. The emotion recognition engine outputs the user's emotional state (e.g., excitement, anxiety).

[0494] Step 4:

[0495] The server generates an appropriate response using a generative AI model based on the recognized emotion. The generative AI model creates an optimal response text based on the user's question content and emotional state. The generative AI model outputs this response text.

[0496] Step 5:

[0497] The server uses the TextToSpeech library to convert the response text into audio data, which is then sent to the device, which plays the audio response, and the user receives the response as audio.

[0498] Step 6:

[0499] After the session ends, the user sends a notification to the server. The server receives the notification and analyzes the session contents, including the user's voice data, text data, and emotional history.

[0500] Step 7:

[0501] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's learning progress and emotional state. The server then sends this feedback report to the terminal and provides it to the user.

[0502] The above are the specific processing steps of the system for realizing the present invention.

[0503] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0505] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0506] [Second embodiment]

[0507] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0508] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0509] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0510] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0511] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0512] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0513] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0514] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0517] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0518] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0519] Based on the problems to be solved by the present invention, specific embodiments for carrying out the invention will be described below.

[0520] 1. Reservation System Implementation

[0521] The user opens an interface on the terminal to reserve a session with an expert. In this interface, the user enters their ID, the desired expert, and the date and time of the session. The terminal sends this reservation information to the server. The server receives the reservation information and stores it in a database. The server then generates a unique reservation ID and returns it to the user.

[0522] 2. Session Format

[0523] On the day of the session, the user asks the expert a question by voice in order to participate in the session. The terminal records the user's voice and sends the voice data to the server. The server analyzes the received voice data and converts it into text using voice recognition technology. The converted text data is input into a generative AI system. The generative AI system generates an appropriate response to the user's question. This response text is again converted into voice by the server and sent to the user's terminal. The terminal plays back the voice data and provides the response to the user.

[0524] 3. Feedback System Embodiments

[0525] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress. It creates a feedback report based on the analysis results and sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[0526] Specific examples

[0527] Consider a specific example where a user wants to book a session with an expert. If User A wants to book a session with Expert B at 3:00 PM on November 1st, User A enters his / her information and desired date and time into the booking interface. The terminal sends this to the server, which stores the booking information in a database, generates a booking ID 12345, and notifies User A.

[0528] On the day of the session, User A voice-inputs the question, "What should I do to learn a new programming language?" The device records the voice and sends it to the server. The server converts the voice data into text and inputs it into the generative AI. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with practical projects." This text response is converted back into speech and provided to User A.

[0529] When the session ends, User A sends a completion notification, and the server analyzes the session content and creates a feedback report. The feedback report includes an evaluation such as "User A understands basic knowledge, but more practical projects would further improve his skills." The user can use this feedback to plan his next learning experience.

[0530] In this way, a system can be implemented that allows users to efficiently improve their skills while receiving guidance from experts.

[0531] The processing flow will be explained below.

[0532] Step 1:

[0533] The user opens an interface on the terminal to reserve a session with an expert, and enters the necessary reservation information, such as the user's ID, the desired expert, and the desired session date and time.

[0534] Step 2:

[0535] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[0536] Step 3:

[0537] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[0538] Step 4:

[0539] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[0540] Step 5:

[0541] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[0542] Step 6:

[0543] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[0544] Step 7:

[0545] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[0546] Step 8:

[0547] The server analyzes the received voice data and converts it into text using speech recognition, which is then input into the conversational AI system.

[0548] Step 9:

[0549] The server uses a conversational AI system to generate appropriate responses to user questions and challenges, and the generated text responses are stored.

[0550] Step 10:

[0551] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[0552] Step 11:

[0553] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[0554] Step 12:

[0555] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[0556] Step 13:

[0557] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio and text data collected during the session.

[0558] Step 14:

[0559] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[0560] Step 15:

[0561] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[0562] Step 16:

[0563] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[0564] Step 17:

[0565] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[0566] Example 1

[0567] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] Conventional systems for booking meetings with experts require a complicated process from reservation to actual meeting, and lack the technology to make it more efficient. Furthermore, evaluating users' learning progress and providing feedback after the meeting is time-consuming, which reduces the overall quality of the service. Another problem is the lack of an established process for efficiently converting voice input into text and generating appropriate responses using artificial intelligence.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0570] In this invention, the server includes: means for a user to reserve a meeting with an expert; means for transmitting the user's reservation information to a communication device and for the communication device to store the reservation information in a storage device; means for providing the user with a generated reservation identifier; means for the communication device to convert voice data into text; means for inputting the generated text data into an artificial intelligence system and for the artificial intelligence system to generate a response; and means for converting the generated response text into voice and providing it to the user. This enables a consistent and efficient process from reserving a meeting to holding it and providing feedback after it has ended.

[0571] "Users" are those who use the system to schedule meetings with experts and ask questions or provide feedback.

[0572] An "expert" is a person from whom a user can meet and receive knowledge and advice.

[0573] A "meeting" is a session in which users and experts interact and ask questions.

[0574] "Reservation information" refers to information that a user enters for a meeting, such as their ID, the desired expert, and the date and time of the meeting.

[0575] A "communication device" is an electronic device that transmits user input data and voice data to a server and conveys the user's response.

[0576] A "storage device" is a storage system that includes a database in which user reservation information and the like are stored.

[0577] A "Reservation Identifier" is a generated unique ID that identifies a particular reservation.

[0578] "Voice data" is digital data of the voice input by the user to ask a question to an expert.

[0579] "Text data" is character information converted from voice data using voice recognition technology.

[0580] An "artificial intelligence system" is a machine learning-based system that generates appropriate responses based on a user's text data.

[0581] "Response text" is a textual response to a user's question generated by an artificial intelligence system.

[0582] "Speech synthesis technology" is a technology for converting text data into voice data.

[0583] An "evaluation report" is a document summarizing the results of analyzing the session content and evaluating the user's learning progress.

[0584] MODE FOR CARRYING OUT THE INVENTION

[0585] The system of the present invention is designed to efficiently and automatically carry out the entire process of a user reserving a meeting with an expert, conducting the meeting, and receiving feedback after the meeting. This system operates mainly through three parties: a server, a terminal, and a user.

[0586] Reservation System Embodiment

[0587] Users can book a meeting with an expert using a dedicated interface. Specifically, users enter their ID, the desired expert, and the date and time of the meeting. This interface is built using libraries such as React and Angular on the front end, Node.js and Django on the back end, and MySQL and PostgreSQL as the database.

[0588] The terminal sends the collected reservation information to the server, which stores the received reservation information in a database, generates a unique reservation identifier (reservation ID), and returns it to the user.

[0589] For example, the user enters "userA, expert B, 2023-11-01 15:00" and the terminal sends the reservation information to the server. The server saves the reservation information in a storage device, generates a reservation ID "12345", and returns it to the user.

[0590] Meeting System Embodiments

[0591] On the day of the meeting, the user speaks their question to the expert. The device records the user's voice data and sends it to the server, which then converts the voice data into text using a speech recognition service (such as Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[0592] The converted text data is then fed into a generative AI model (such as OpenAI GPT-4 or Google BERT) to generate an appropriate response. The generated response text is then converted into audio data using speech synthesis technology and sent to the user's device. The device then plays back the audio data and provides it to the user.

[0593] For example, a user may ask, "What should I do to learn a new programming language?" The server converts the voice data into text, and the generative AI generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This response is then converted into audio and provided to the user.

[0594] Feedback System Embodiments

[0595] After the meeting, the user sends a notification to the system. The device then sends this notification to the server, which then analyzes the session content and evaluates the user's learning progress. Machine learning techniques can be used for the analysis.

[0596] The server then creates a feedback report based on the evaluation results and sends it to the user, who can review the feedback report and use it to plan their next study.

[0597] As a concrete example, a user sends a completion notification to the server, and the server generates a feedback report stating, "User A understands basic knowledge, but his skills will improve further with more practical projects," and sends it to the user.

[0598] Prompt Sentence Examples

[0599] Please provide a detailed explanation, using examples, of how a user schedules a meeting with an expert, conducts the meeting, and receives feedback. Please also include specific actions taken by the server, device, and user during this process.

[0600] This system allows users to efficiently go through a series of processes while receiving expert guidance.

[0601] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0602] Specific explanation of processing steps

[0603] Reservation System

[0604] Step 1: View the interface

[0605] The terminal displays an interface for the user to schedule a meeting with an expert, the interface including fields for entering the user ID, the desired expert, and the date and time of the meeting.

[0606] input

[0607] User operation (interface display request)

[0608] output

[0609] View the booking interface

[0610] Specific actions

[0611] The terminal loads the reservation interface HTML page and displays it to the user.

[0612] Step 2: Enter reservation information

[0613] The user enters their ID, the desired expert, and the date and time of the meeting into the interface.

[0614] input

[0615] User input information (ID, expert, date and time)

[0616] output

[0617] Reservation information entered

[0618] Specific actions

[0619] The user enters information using a keyboard or touch screen.

[0620] Step 3: Submit your reservation information

[0621] The terminal transmits the reservation information input by the user to the server.

[0622] input

[0623] Reservation information (ID, specialist, date and time)

[0624] output

[0625] Sending data to the server

[0626] Specific actions

[0627] The terminal sends the reservation information to the server as an HTTP request.

[0628] Step 4: Save your reservation

[0629] The server stores the received reservation information in a database.

[0630] input

[0631] Received reservation information

[0632] output

[0633] Reservation information recorded in the database

[0634] Specific actions

[0635] The server converts the reservation information into the appropriate format and stores it in a database using SQL.

[0636] Step 5: Generate a reservation ID

[0637] The server generates a unique reservation identifier (reservation ID).

[0638] input

[0639] Saved reservations

[0640] output

[0641] Generated reservation ID

[0642] Specific actions

[0643] The server runs an algorithm to generate a unique ID based on the reservation information.

[0644] Step 6: Reservation ID response

[0645] The server returns the generated reservation ID to the user.

[0646] input

[0647] Generated reservation ID

[0648] output

[0649] Sending the reservation ID to the user

[0650] Specific actions

[0651] The server creates an HTTP response including the reservation ID and sends it to the terminal.

[0652] Meeting System

[0653] Step 1: Enter your voice question

[0654] On the day of the session, the user will use the terminal to voice-type questions to the expert.

[0655] input

[0656] User voice questions

[0657] output

[0658] Audio data

[0659] Specific actions

[0660] The user speaks a question into the microphone.

[0661] Step 2: Sending audio data

[0662] The terminal records the user's voice and transmits the voice data to the server.

[0663] input

[0664] Audio data

[0665] output

[0666] Sending audio data to the server

[0667] Specific actions

[0668] The device records the audio and sends it to the server in WAV or MP3 format.

[0669] Step 3: Convert audio data to text

[0670] The server converts the received voice data into text using voice recognition technology.

[0671] input

[0672] Audio data

[0673] output

[0674] Text data

[0675] Specific actions

[0676] The server calls a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the speech into text.

[0677] Step 4: AI input of text data

[0678] The server inputs the converted text data into a generative AI model.

[0679] input

[0680] Text data

[0681] output

[0682] Input data for AI models

[0683] Specific actions

[0684] The server converts the text data into a format suitable for the AI ​​model and sends it to the model.

[0685] Step 5: Generate an AI response

[0686] Generative AI models (e.g., OpenAI GPT-4) generate appropriate responses to user questions.

[0687] input

[0688] Input data for AI models

[0689] output

[0690] Response Text

[0691] Specific actions

[0692] The AI ​​model performs the calculations to generate an answer to the question.

[0693] Step 6: Convert response text to speech

[0694] The server converts the generated response text into voice data using voice synthesis technology.

[0695] input

[0696] Response Text

[0697] output

[0698] Audio data

[0699] Specific actions

[0700] The server calls a speech synthesis API (e.g., Amazon Polly) to convert the text to speech.

[0701] Step 7: Sending a response audio

[0702] The server transmits the generated voice data to the user's terminal.

[0703] input

[0704] Audio data

[0705] output

[0706] Sending audio to the user

[0707] Specific actions

[0708] The server sends the audio data to the terminal as an HTTP response.

[0709] Step 8: Play the audio response

[0710] The terminal plays this audio data and provides a response to the user.

[0711] input

[0712] Audio data

[0713] output

[0714] Playing audio

[0715] Specific actions

[0716] The device plays the audio data using a playback application.

[0717] Feedback System

[0718] Step 1: Sending a termination notice

[0719] After the meeting is over, the user sends a notification of the end to the system.

[0720] input

[0721] Termination Notice

[0722] output

[0723] Sending notifications to the server

[0724] Specific actions

[0725] The user clicks the end meeting button and a notification is sent to the server.

[0726] Step 2: Receive notifications

[0727] The terminal sends a termination notification to the server.

[0728] input

[0729] Termination Notice

[0730] output

[0731] Sending notifications to the server

[0732] Specific actions

[0733] The terminal sends a termination notification to the server as an HTTP request.

[0734] Step 3: Analyzing session content

[0735] The server analyzes the session content based on the received notification and evaluates the user's learning progress.

[0736] input

[0737] End notice and session details

[0738] output

[0739] Evaluation Data

[0740] Specific actions

[0741] The server analyzes the session data using machine learning algorithms.

[0742] Step 4: Create a feedback report

[0743] The server generates a feedback report based on the analysis results.

[0744] input

[0745] Evaluation Data

[0746] output

[0747] Feedback Report

[0748] Specific actions

[0749] The server runs an algorithm that automatically generates a feedback report.

[0750] Step 5: Submit the report

[0751] The server sends the generated feedback report to the user.

[0752] input

[0753] Feedback Report

[0754] output

[0755] Sending reports to users

[0756] Specific actions

[0757] The server sends the feedback report as an HTTP response.

[0758] Step 6: Review the report

[0759] Users can check the feedback report and use it to plan their future studies.

[0760] input

[0761] Feedback Report

[0762] output

[0763] Verified Feedback

[0764] Specific actions

[0765] The user displays the feedback report on the device and checks the contents.

[0766] (Application example 1)

[0767] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0768] Current factory operations often rely on human resources for troubleshooting and work guidance, which can lead to reduced productivity and the need for rapid response when problems occur. Furthermore, it can take time for staff to learn new technologies and operating procedures, making it difficult for them to respond immediately. Therefore, there is a need for a system that can provide real-time problem solving and guidance through a voice interface.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0770] In this invention, the server includes a means for recognizing user voice data and converting it into text data, a means for inputting the text data into a generating artificial intelligence, which then generates a response, and a means for providing the generated voice response to the user, thereby enabling quick troubleshooting and work guidance within the factory.

[0771] "User" refers to the person or factory staff who uses the system.

[0772] An "expert" is a person or system with advanced specialized knowledge or skills.

[0773] A "session" is a period of time during which information is exchanged or consulted between a user and an expert.

[0774] A "reservation" is a prior application or registration to use a system or expert at a specific date and time.

[0775] A "server" is a computer system that processes reservation information and voice data and inputs them into the generation AI.

[0776] "Voice data" refers to data in which a user's questions or requests are recorded in the form of sound.

[0777] "Text data" refers to data obtained by converting voice data into character information.

[0778] "Generative artificial intelligence" refers to an AI system that generates appropriate responses based on input text data.

[0779] "Response text" refers to data that expresses the answer content generated by the generating AI in text.

[0780] A "feedback report" is a report that summarizes the user's learning progress and areas for improvement after a session.

[0781] System program description

[0782] The system for realizing this invention is constructed using the following hardware and software: a user's terminal, a server, a generative AI model, speech recognition technology, and speech synthesis technology.

[0783] Hardware and Software Use

[0784] 1. On the user's device:

[0785] Microphone (audio input)

[0786] Speaker (audio output)

[0787] Internet connection function (data communication)

[0788] 2. Server:

[0789] Database (storing reservation information)

[0790] Speech recognition technology (converts voice to text)

[0791] Generative AI models (generate responses from text)

[0792] Speech synthesis technology (converting text to speech)

[0793] Details of data processing and calculation

[0794] The server receives the user's voice data and converts it into text using speech recognition technology, a process that uses software such as speech_recognition.

[0795] The converted text data is then fed into a generative artificial intelligence system, which uses a generative AI model such as GPT-2 to generate an optimal response based on the input text.

[0796] The generated text response is converted into voice data using speech synthesis technology, using software such as pyttsx3.

[0797] Finally, this voice data is sent to the user's terminal and presented to the user through a speaker.

[0798] Specific examples

[0799] Reservation System:

[0800] Example: When a user books a session with an expert, they enter their information and desired date and time in the terminal interface. The information is sent to the server and stored in a database. The server generates a unique booking ID and sends it back to the user.

[0801] session:

[0802] Example: On the day of the session, the user voice-inputs the question, "What should I do to learn a new programming language?" The server converts the voice data into text and feeds it into a generative AI model. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This text response is converted into speech and provided to the user's device.

[0803] Prompt Sentence Examples

[0804] Here is an example prompt for the example above:

[0805] Generate an appropriate response to "What caused this machine in the factory to stop working?"

[0806] This will enable quick troubleshooting and work guidance within the factory, and is expected to improve productivity.

[0807] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0808] Step 1:

[0809] The user opens the reservation interface on the terminal to reserve a session with an expert. The user enters their information (name and ID), the desired expert, and the date and time of the session. The terminal sends the reservation information entered by the user to the server. The server receives the reservation information and stores it in a database. The server generates a unique reservation ID and returns it to the user via the terminal.

[0810] Step 2:

[0811] On the day of the session, the user uses the microphone on the device to ask the expert a question. The device records the user's voice data and sends it to the server. The input at this time is the user's voice data.

[0812] Step 3:

[0813] The server converts the received voice data into text data using speech recognition software. This process uses the speech_recognition software. By converting the voice data into text, the content of the user's question is output as text data.

[0814] Step 4:

[0815] The server inputs the user's question, converted into text, into a generative AI model. GPT-2 is used as the generative AI model. The text data is passed as input to the AI ​​model, which generates an appropriate response text. The generative AI model generates a response using a prompt sentence based on the user's question. Specifically, a response based on the prompt sentence, "Please generate an appropriate reply to the question 'What caused this machine in the factory to stop?'" is output.

[0816] Step 5:

[0817] The generated response text is converted into voice data by the speech synthesis system in the server. For this purpose, we use the pyttsx3 software, which generates voice data from the text data and outputs the response voice.

[0818] Step 6:

[0819] The server transmits the generated voice data to the terminal, which receives the voice data and provides a response to the user through a speaker, where the output is a voice response that the user can hear.

[0820] Step 7:

[0821] After the session ends, the user sends a notification to the server via their device. The server receives this notification, analyzes the session, and creates a feedback report. This analysis and feedback includes usage data, user questions, and AI responses.

[0822] Step 8:

[0823] The server sends the created feedback report to the terminal, which displays the report to the user, allowing the user to use it for future learning and work improvement. The output at this time is the feedback report.

[0824] The above process realizes a system that allows users to quickly receive expert guidance and troubleshooting suggestions in real time using voice.

[0825] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0826] Based on the problem to be solved by the present invention, a specific embodiment for carrying out the invention in combination with an emotion engine will be described below.

[0827] 1. Reservation System Implementation

[0828] The user opens an interface on their terminal to reserve a session with an expert and enters the necessary reservation information, such as their ID, desired expert, and desired session date and time. The terminal sends this reservation information to the server's reservation API via a POST request. The server receives the reservation information and generates a unique reservation ID when saving it in the database. The server then notifies the user of the generated reservation ID.

[0829] 2. Session Format

[0830] At the start of a session, the user inputs a question or task by voice. The device records the user's voice and sends the voice data to the server. The server analyzes the received voice data, converts it into text using voice recognition technology, and then recognizes the user's emotions using an emotion engine. Based on the recognized emotions, the dialogue AI system generates an appropriate response. The generated text response is converted back into voice by the server and sent to the device. The device plays back this voice data and provides the response to the user.

[0831] 3. Feedback System Embodiments

[0832] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[0833] Specific examples using emotion engines

[0834] Consider a scenario where, during a session, the user asks a question by voice, such as, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes the text data to the emotion engine. The emotion engine analyzes the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, the response generated might be, "Start with a basic tutorial, then hone your skills with practical projects." The server then converts this text back into audio data and provides it to the user.

[0835] Dynamic Adjustment Examples

[0836] The server evaluates the user's emotions in real time during the session and dynamically adjusts the expert's instruction. For example, if the user shows signs of anxiety, the expert will flexibly change the instruction to suit the situation. This allows the user to learn in a relaxed state and effectively acquire skills.

[0837] Example of a feedback report

[0838] After the session ends, the server receives a notification and analyzes the session content. This analysis includes the user's voice data, text data, and emotional history. Based on the analysis results, the server provides feedback such as, "The user has grasped basic knowledge, and in the future, more practical projects will further improve their skills." The feedback report is sent to the user, who can use it to plan their next learning.

[0839] In this way, a system can be implemented in which users can smoothly progress through learning by utilizing the emotion engine while receiving guidance from an expert, and can efficiently improve their skills.

[0840] The processing flow will be explained below.

[0841] Step 1:

[0842] The user opens an interface on the terminal for reserving a session with an expert and inputs necessary reservation information such as the user's ID, the desired expert, and the desired date and time of the session.

[0843] Step 2:

[0844] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[0845] Step 3:

[0846] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[0847] Step 4:

[0848] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[0849] Step 5:

[0850] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[0851] Step 6:

[0852] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[0853] Step 7:

[0854] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[0855] Step 8:

[0856] The server analyzes the received voice data and converts the voice data into text using speech recognition, which is then input to the emotion engine.

[0857] Step 9:

[0858] The server uses an emotion engine to recognize the user's emotions from their voice, and classifies the emotions into categories such as "happiness," "sadness," "anger," and "surprise."

[0859] Step 10:

[0860] The server inputs text data based on the user's emotions into the conversational AI system, which then generates an appropriate response. For example, if the user expresses "anxiety," the AI ​​will generate a more friendly and reassuring response.

[0861] Step 11:

[0862] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[0863] Step 12:

[0864] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[0865] Step 13:

[0866] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[0867] Step 14:

[0868] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio data, text data, and emotion history collected during the session.

[0869] Step 15:

[0870] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[0871] Step 16:

[0872] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[0873] Step 17:

[0874] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[0875] Step 18:

[0876] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[0877] Example 2

[0878] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0879] Conventional reservation systems have the drawback of requiring users to perform many manual operations when reserving a session with an expert, and making it difficult to manage reservation information. Furthermore, they are unable to grasp the user's emotions in real time during the session and dynamically adjust the expert's instruction content based on those emotions, resulting in insufficient learning effectiveness. Furthermore, feedback after the session is sometimes delayed, making it difficult to adequately support the user's learning progress.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0881] In this invention, the server includes means for transmitting the user's reservation information to the server and saving the reservation information in a database, means for generating a specific reservation number and providing it to the user, means for transmitting voice data from the user to the server, means for converting the voice data to text, means for inputting the text data into an emotion analysis engine and recognizing emotions, means for generating a response using a generative AI model based on the recognized emotions, means for converting the generated response text into voice and providing it to the user, means for receiving a session end notification from the user and analyzing the session content and the user's emotion history to make a final evaluation, and means for creating a feedback report and sending it to the user. This enables the user to make a reservation smoothly, receive appropriate responses in real time during the session, and receive prompt feedback after the session ends.

[0882] A "user" is a person who utilizes the system to book and conduct sessions with experts.

[0883] An "expert" is someone who has extensive knowledge and experience in a particular field and who can provide guidance and advice to users.

[0884] A "server" is a computer system that receives information from users, processes the data, stores it in a database, and runs generative AI models.

[0885] A "terminal" is a device used by a user to access the system, and includes a PC, smartphone, tablet, etc.

[0886] A "generative AI model" is an artificial intelligence algorithm that generates optimal responses or predictions based on specific input data.

[0887] A "reservation number" is a unique identifier that allows a user to identify a reservation.

[0888] "Voice data" refers to data in which the user's voice is recorded in digital format.

[0889] "Text data" refers to character data obtained by analyzing and transcribing voice data.

[0890] An "emotion analysis engine" is an analysis device or program that analyzes a user's emotions (e.g., excitement, anxiety, joy, etc.) from input text data.

[0891] A "feedback report" is a report that includes an evaluation and suggestions for future learning, generated based on the user's performance and emotional history during the session.

[0892] This invention is a system that allows users to reserve sessions with experts, and receive feedback on the sessions. This system is composed of three entities: a server, a terminal, and a user, and operates as follows.

[0893] 1. Reservation System Implementation

[0894] To book a session with an expert, a user opens the device's interface and enters the necessary information, such as their ID, the preferred expert, and the desired date and time of the session. This interface is provided as a web page or mobile app. The device sends this reservation information to the server's reservation API via an HTTP POST request. The server executes an SQL query to store the received reservation information in a database and generates a unique reservation number. The server then returns a response including the generated reservation number to the device, and the user is notified. This allows the user to confirm that the reservation was successful.

[0895] 2. Session Format

[0896] At the start of a session, the user uses the voice input function to speak a question or task into the device. The device records the user's voice and sends the audio data to the server. This audio data is saved in an appropriate format (such as .wav or .mp3). The server analyzes the received audio data and converts it into text using speech recognition technology (e.g., Google Cloud Speech-to-Text). The server then passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. Based on the recognized emotion, a generative AI model (e.g., OpenAI GPT-3) generates a response. This response is again converted into speech by the server (e.g., Amazon Polly) and sent to the device. The device plays the audio data and provides the response to the user.

[0897] 3. Feedback System Embodiments

[0898] After the session ends, the user sends a completion notification to the system. The device then sends this completion notification to the server. The server analyzes the session content based on the received completion notification (e.g., using an NLP analysis tool) and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning.

[0899] Add specific examples to the description

[0900] Reservation system example

[0901] For example, if a user wants to reserve a session with an expert "Expert A" with ID "user123" for 15:00 on November 1, 2023, he or she enters this information into the terminal interface. The terminal sends this information to the server, which stores it in its database, generates a reservation number "reserve456" and notifies the user.

[0902] Session Example

[0903] For example, if a user asks, "What should I do to learn a new programming language?", the device records the question and sends it to a server. The server converts the speech to text using Google Cloud Speech-to-Text and analyzes sentiment using IBM Watson Tone Analyzer. OpenAI GPT-3 generates a response such as "Start with basic tutorials and hone your skills with hands-on projects," and Amazon Polly converts this text to speech and serves it to the user.

[0904] Examples of feedback systems

[0905] After the session ends, the server receives a notification from the user and analyzes the user's emotional history and session data. It then generates feedback to the user, such as "The user has grasped basic knowledge and would benefit from more practical projects."

[0906] Prompt Sentence Examples

[0907] "What should I do when learning a new programming language?"

[0908] Through each of these steps, users can enjoy a smooth booking, session, and feedback experience, and a system can be implemented that allows them to efficiently improve their skills while receiving expert guidance.

[0909] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0910] Step 1:

[0911] The user opens the device interface and enters information such as their ID, the preferred specialist, and the desired date and time of the session. This information is entered into the device as reservation information. The device temporarily stores this information and displays a confirmation screen to the user.

[0912] Step 2:

[0913] When the user confirms the reservation information and presses the send button, the device sends this reservation information to the server's reservation API via an HTTP POST request. The input data is the user ID, expert ID, and reservation date and time, and is sent to the server in JSON format.

[0914] Step 3:

[0915] The server analyzes the reservation information received from the terminal and generates an SQL query to store it in the database. Here, the user ID, expert ID, and reservation date and time are input data, and are stored in fields in the "Reservation" table. The server generates a unique reservation number and returns it as a response.

[0916] Step 4:

[0917] The terminal receives the response from the server and displays a confirmation message to the user, including the reservation number. This confirms that the reservation was successfully completed. The output is the reservation number and the confirmation message.

[0918] Step 5:

[0919] At the scheduled time, the user starts the session using the device. Using the voice input function, the user voice-enters questions and tasks into the device. This voice data becomes the input data.

[0920] Step 6:

[0921] The device records the user's voice and sends it to the server. The voice data is encoded in the appropriate format (e.g., .wav or .mp3). The data sent to the server is the binary data of the voice file.

[0922] Step 7:

[0923] The server analyzes the received voice data and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The input data is an audio file, and the output is text data. The server temporarily stores the text data.

[0924] Step 8:

[0925] The server passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data, and the output is the emotion recognition result (e.g., excitement, anxiety, etc.). The server temporarily stores the emotion recognition result.

[0926] Step 9:

[0927] The server inputs the recognized emotion and text data into a generative AI model (e.g., OpenAI GPT-3) to generate an appropriate response. The input data is text and emotion recognition results, and the output is a response text. The server temporarily stores the generated response text.

[0928] Step 10:

[0929] The server converts the generated response text into speech using speech synthesis technology (e.g., Amazon Polly). The input data is the response text, and the output is an audio file. The server then sends the audio data to the device.

[0930] Step 11:

[0931] The terminal plays back the voice data received from the server and provides a response to the user. The output is a voice response, allowing the user to experience a dialogue in real time.

[0932] Step 12:

[0933] After the session ends, the user sends a termination notification to the system. By clicking the end button, the terminal sends a termination notification to the server. The data sent is a termination notification flag.

[0934] Step 13:

[0935] The server receives the termination notification and analyzes the audio data, text data, and emotion history during the session (e.g., using an NLP analysis tool). The input data are audio data, text data, and emotion recognition results, and the output is the analysis results.

[0936] Step 14:

[0937] The server generates a feedback report based on the analysis results. The input data is the analysis results, and the output is the feedback report. The report contains information that is useful for the user's learning progress and future learning plans.

[0938] Step 15:

[0939] The server sends a feedback report to the user, and the sent data is the feedback report, which the user can check and use for future learning.

[0940] (Application example 2)

[0941] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0942] Conventional online session systems lack support for users to communicate effectively with experts. Furthermore, because they respond uniformly without considering the user's emotions, it is difficult to provide appropriate support for emotions such as anxiety and excitement. Furthermore, because feedback does not take into account the user's emotions or learning progress, it is difficult to create an effective learning plan.

[0943] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing user reservation information in a database, means for analyzing the user's voice input in real time based on the generated reservation information and recognizing emotions, means for generating an appropriate response to the voice input based on the emotion recognition, and means for dynamically adjusting and reproducing the response taking the user's emotions into consideration. This makes it possible to grasp the user's emotions in real time and provide an optimal response. In addition, by evaluating the user's learning progress and providing appropriate feedback, it is possible to create an effective learning plan.

[0944] "User" refers to a person who uses the system to hold a session with an expert.

[0945] An "expert" refers to a person or system that has specialized knowledge and skills in a particular field and that holds a session with a user to provide assistance.

[0946] A "session" refers to a period of interaction or instruction between a user and an expert for a specific purpose.

[0947] "Reservation information" refers to information provided by a user when reserving a session with an expert, and includes, for example, an ID, a desired expert, and a desired date and time for the session.

[0948] "Server" refers to a central processing unit that processes data and communicates between users and experts.

[0949] "Database" refers to a system that centrally manages and stores data such as reservation information.

[0950] A "reservation ID" is a unique identifier generated by the server and used to manage session reservations.

[0951] "Analyzing in real time" refers to a process of instantly analyzing voice data while the user is inputting voice data.

[0952] "Emotion recognition" refers to analyzing and identifying emotional states such as excitement, anxiety, and joy from a user's voice and text data.

[0953] "Dynamic response adjustment" refers to changing and optimizing expert guidance and system responses in real time according to the user's emotional state.

[0954] A "feedback report" is a report generated by the server after a session ends. It evaluates the user's learning progress and emotional history, and provides information that can be used as a reference for future learning plans.

[0955] The present invention provides a product purchasing support system in a virtual store that allows users to reserve a session with an expert and dynamically responds to user emotions. Specific embodiments of this system are described below.

[0956] 1. Reservation System Implementation

[0957] A user uses their device (e.g., smartphone, tablet, or computer) to reserve a session with an expert. The user opens the reservation interface and enters the necessary information, such as their ID, the desired expert, and the desired session date and time. The device then sends this reservation information to the server's reservation API via a POST request. The server stores the received reservation information in a database and generates a unique reservation ID. This reservation ID is notified to the user.

[0958] 2. Session Format

[0959] At the start of a session, the user vocally inputs a question or task related to the session content. The device records the user's voice and sends the voice data to the server. The server then uses speech recognition technology (e.g., Google Speech Recognition API) to convert the voice data into text. It then uses an emotion recognition engine (e.g., EmotionRecognizer library) to analyze and recognize the user's emotions from the text data. Based on the recognized emotions, the server's conversational AI system (e.g., generative AI model) generates an appropriate response. The generated text response is converted into voice data by the server (e.g., TextToSpeech library), sent to the device, and provided to the user.

[0960] 3. Feedback System Embodiments

[0961] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning. In this way, the user can effectively progress with their learning by utilizing the emotion engine while receiving guidance from an expert.

[0962] Specific examples

[0963] For example, consider a case where a user asks by voice, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes this text data to an emotion recognition engine. The emotion recognition engine analyzes emotions from the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, it might generate a response as text, such as "Start with a basic tutorial and then hone your skills with practical projects," and the server then converts this text back into audio data and provides it to the user.

[0964] Prompt Sentence Examples

[0965] User Question: "What should I do when learning a new programming language?"

[0966] Recognized emotion: "excitement"

[0967] This allows users to receive expert guidance in a way that suits their emotions, resulting in a more effective and personalized purchasing experience and learning.

[0968] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0969] Step 1:

[0970] The user opens the reservation interface and enters reservation information such as ID, desired expert, desired session date and time, etc. The terminal sends this input information to the server. The server stores the received reservation information in a database and generates a unique reservation ID. The server notifies the user of this reservation ID.

[0971] Step 2:

[0972] At the start of a session, the user enters a question or task by voice through the device. The device records the user's voice and sends the voice data to the server. The server receives the voice data and converts it into text using voice recognition technology.

[0973] Step 3:

[0974] The server inputs the converted text data into an emotion recognition engine, which analyzes the text data and identifies the user's emotion. The emotion recognition engine outputs the user's emotional state (e.g., excitement, anxiety).

[0975] Step 4:

[0976] The server generates an appropriate response using a generative AI model based on the recognized emotion. The generative AI model creates an optimal response text based on the user's question content and emotional state. The generative AI model outputs this response text.

[0977] Step 5:

[0978] The server uses the TextToSpeech library to convert the response text into audio data, which is then sent to the device, which plays the audio response, and the user receives the response as audio.

[0979] Step 6:

[0980] After the session ends, the user sends a notification to the server. The server receives the notification and analyzes the session contents, including the user's voice data, text data, and emotional history.

[0981] Step 7:

[0982] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's learning progress and emotional state. The server then sends this feedback report to the terminal and provides it to the user.

[0983] The above are the specific processing steps of the system for realizing the present invention.

[0984] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0985] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0986] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0987] [Third embodiment]

[0988] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0989] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0990] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0991] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0992] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0993] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0994] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0995] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0996] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0997] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0998] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0999] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1000] Based on the problems to be solved by the present invention, specific embodiments for carrying out the invention will be described below.

[1001] 1. Reservation System Implementation

[1002] The user opens an interface on the terminal to reserve a session with an expert. In this interface, the user enters their ID, the desired expert, and the date and time of the session. The terminal sends this reservation information to the server. The server receives the reservation information and stores it in a database. The server then generates a unique reservation ID and returns it to the user.

[1003] 2. Session Format

[1004] On the day of the session, the user asks the expert a question by voice in order to participate in the session. The terminal records the user's voice and sends the voice data to the server. The server analyzes the received voice data and converts it into text using voice recognition technology. The converted text data is input into a generative AI system. The generative AI system generates an appropriate response to the user's question. This response text is again converted into voice by the server and sent to the user's terminal. The terminal plays back the voice data and provides the response to the user.

[1005] 3. Feedback System Embodiments

[1006] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress. It creates a feedback report based on the analysis results and sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[1007] Specific examples

[1008] Consider a specific example where a user wants to book a session with an expert. If User A wants to book a session with Expert B at 3:00 PM on November 1st, User A enters his / her information and desired date and time into the booking interface. The terminal sends this to the server, which stores the booking information in a database, generates a booking ID 12345, and notifies User A.

[1009] On the day of the session, User A voice-inputs the question, "What should I do to learn a new programming language?" The device records the voice and sends it to the server. The server converts the voice data into text and inputs it into the generative AI. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with practical projects." This text response is converted back into speech and provided to User A.

[1010] When the session ends, User A sends a completion notification, and the server analyzes the session content and creates a feedback report. The feedback report includes an evaluation such as "User A understands basic knowledge, but more practical projects would further improve his skills." The user can use this feedback to plan his next learning experience.

[1011] In this way, a system can be implemented that allows users to efficiently improve their skills while receiving guidance from experts.

[1012] The processing flow will be explained below.

[1013] Step 1:

[1014] The user opens an interface on the terminal to reserve a session with an expert, and enters the necessary reservation information, such as the user's ID, the desired expert, and the desired session date and time.

[1015] Step 2:

[1016] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[1017] Step 3:

[1018] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[1019] Step 4:

[1020] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[1021] Step 5:

[1022] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[1023] Step 6:

[1024] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[1025] Step 7:

[1026] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[1027] Step 8:

[1028] The server analyzes the received voice data and converts it into text using speech recognition, which is then input into the conversational AI system.

[1029] Step 9:

[1030] The server uses a conversational AI system to generate appropriate responses to user questions and challenges, and the generated text responses are stored.

[1031] Step 10:

[1032] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[1033] Step 11:

[1034] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[1035] Step 12:

[1036] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[1037] Step 13:

[1038] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio and text data collected during the session.

[1039] Step 14:

[1040] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[1041] Step 15:

[1042] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[1043] Step 16:

[1044] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[1045] Step 17:

[1046] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[1047] Example 1

[1048] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1049] Conventional systems for booking meetings with experts require a complicated process from reservation to actual meeting, and lack the technology to make it more efficient. Furthermore, evaluating users' learning progress and providing feedback after the meeting is time-consuming, which reduces the overall quality of the service. Another problem is the lack of an established process for efficiently converting voice input into text and generating appropriate responses using artificial intelligence.

[1050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1051] In this invention, the server includes: means for a user to reserve a meeting with an expert; means for transmitting the user's reservation information to a communication device and for the communication device to store the reservation information in a storage device; means for providing the user with a generated reservation identifier; means for the communication device to convert voice data into text; means for inputting the generated text data into an artificial intelligence system and for the artificial intelligence system to generate a response; and means for converting the generated response text into voice and providing it to the user. This enables a consistent and efficient process from reserving a meeting to holding it and providing feedback after it has ended.

[1052] "Users" are those who use the system to schedule meetings with experts and ask questions or provide feedback.

[1053] An "expert" is a person from whom a user can meet and receive knowledge and advice.

[1054] A "meeting" is a session in which users and experts interact and ask questions.

[1055] "Reservation information" refers to information that a user enters for a meeting, such as their ID, the desired expert, and the date and time of the meeting.

[1056] A "communication device" is an electronic device that transmits user input data and voice data to a server and conveys the user's response.

[1057] A "storage device" is a storage system that includes a database in which user reservation information and the like are stored.

[1058] A "Reservation Identifier" is a generated unique ID that identifies a particular reservation.

[1059] "Voice data" is digital data of the voice input by the user to ask a question to an expert.

[1060] "Text data" is character information converted from voice data using voice recognition technology.

[1061] An "artificial intelligence system" is a machine learning-based system that generates appropriate responses based on a user's text data.

[1062] "Response text" is a textual response to a user's question generated by an artificial intelligence system.

[1063] "Speech synthesis technology" is a technology for converting text data into voice data.

[1064] An "evaluation report" is a document summarizing the results of analyzing the session content and evaluating the user's learning progress.

[1065] MODE FOR CARRYING OUT THE INVENTION

[1066] The system of the present invention is designed to efficiently and automatically carry out the entire process of a user reserving a meeting with an expert, conducting the meeting, and receiving feedback after the meeting. This system operates mainly through three parties: a server, a terminal, and a user.

[1067] Reservation System Embodiment

[1068] Users can book a meeting with an expert using a dedicated interface. Specifically, users enter their ID, the desired expert, and the date and time of the meeting. This interface is built using libraries such as React and Angular on the front end, Node.js and Django on the back end, and MySQL and PostgreSQL as the database.

[1069] The terminal sends the collected reservation information to the server, which stores the received reservation information in a database, generates a unique reservation identifier (reservation ID), and returns it to the user.

[1070] For example, the user enters "userA, expert B, 2023-11-01 15:00" and the terminal sends the reservation information to the server. The server saves the reservation information in a storage device, generates a reservation ID "12345", and returns it to the user.

[1071] Meeting System Embodiments

[1072] On the day of the meeting, the user speaks their question to the expert. The device records the user's voice data and sends it to the server, which then converts the voice data into text using a speech recognition service (such as Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[1073] The converted text data is then fed into a generative AI model (such as OpenAI GPT-4 or Google BERT) to generate an appropriate response. The generated response text is then converted into audio data using speech synthesis technology and sent to the user's device. The device then plays back the audio data and provides it to the user.

[1074] For example, a user may ask, "What should I do to learn a new programming language?" The server converts the voice data into text, and the generative AI generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This response is then converted into audio and provided to the user.

[1075] Feedback System Embodiments

[1076] After the meeting, the user sends a notification to the system. The device then sends this notification to the server, which then analyzes the session content and evaluates the user's learning progress. Machine learning techniques can be used for the analysis.

[1077] The server then creates a feedback report based on the evaluation results and sends it to the user, who can review the feedback report and use it to plan their next study.

[1078] As a concrete example, a user sends a completion notification to the server, and the server generates a feedback report stating, "User A understands basic knowledge, but his skills will improve further with more practical projects," and sends it to the user.

[1079] Prompt Sentence Examples

[1080] Please provide a detailed explanation, using examples, of how a user schedules a meeting with an expert, conducts the meeting, and receives feedback. Please also include specific actions taken by the server, device, and user during this process.

[1081] This system allows users to efficiently go through a series of processes while receiving expert guidance.

[1082] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1083] Specific explanation of processing steps

[1084] Reservation System

[1085] Step 1: View the interface

[1086] The terminal displays an interface for the user to schedule a meeting with an expert, the interface including fields for entering the user ID, the desired expert, and the date and time of the meeting.

[1087] input

[1088] User operation (interface display request)

[1089] output

[1090] View the booking interface

[1091] Specific actions

[1092] The terminal loads the reservation interface HTML page and displays it to the user.

[1093] Step 2: Enter reservation information

[1094] The user enters their ID, the desired expert, and the date and time of the meeting into the interface.

[1095] input

[1096] User input information (ID, expert, date and time)

[1097] output

[1098] Reservation information entered

[1099] Specific actions

[1100] The user enters information using a keyboard or touch screen.

[1101] Step 3: Submit your reservation information

[1102] The terminal transmits the reservation information input by the user to the server.

[1103] input

[1104] Reservation information (ID, specialist, date and time)

[1105] output

[1106] Sending data to the server

[1107] Specific actions

[1108] The terminal sends the reservation information to the server as an HTTP request.

[1109] Step 4: Save your reservation

[1110] The server stores the received reservation information in a database.

[1111] input

[1112] Received reservation information

[1113] output

[1114] Reservation information recorded in the database

[1115] Specific actions

[1116] The server converts the reservation information into the appropriate format and stores it in a database using SQL.

[1117] Step 5: Generate a reservation ID

[1118] The server generates a unique reservation identifier (reservation ID).

[1119] input

[1120] Saved reservations

[1121] output

[1122] Generated reservation ID

[1123] Specific actions

[1124] The server runs an algorithm to generate a unique ID based on the reservation information.

[1125] Step 6: Reservation ID response

[1126] The server returns the generated reservation ID to the user.

[1127] input

[1128] Generated reservation ID

[1129] output

[1130] Sending the reservation ID to the user

[1131] Specific actions

[1132] The server creates an HTTP response including the reservation ID and sends it to the terminal.

[1133] Meeting System

[1134] Step 1: Enter your voice question

[1135] On the day of the session, the user will use the terminal to voice-type questions to the expert.

[1136] input

[1137] User voice questions

[1138] output

[1139] Audio data

[1140] Specific actions

[1141] The user speaks a question into the microphone.

[1142] Step 2: Sending audio data

[1143] The terminal records the user's voice and transmits the voice data to the server.

[1144] input

[1145] Audio data

[1146] output

[1147] Sending audio data to the server

[1148] Specific actions

[1149] The device records the audio and sends it to the server in WAV or MP3 format.

[1150] Step 3: Convert audio data to text

[1151] The server converts the received voice data into text using voice recognition technology.

[1152] input

[1153] Audio data

[1154] output

[1155] Text data

[1156] Specific actions

[1157] The server calls a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the speech into text.

[1158] Step 4: AI input of text data

[1159] The server inputs the converted text data into a generative AI model.

[1160] input

[1161] Text data

[1162] output

[1163] Input data for AI models

[1164] Specific actions

[1165] The server converts the text data into a format suitable for the AI ​​model and sends it to the model.

[1166] Step 5: Generate an AI response

[1167] Generative AI models (e.g., OpenAI GPT-4) generate appropriate responses to user questions.

[1168] input

[1169] Input data for AI models

[1170] output

[1171] Response Text

[1172] Specific actions

[1173] The AI ​​model performs the calculations to generate an answer to the question.

[1174] Step 6: Convert response text to speech

[1175] The server converts the generated response text into voice data using voice synthesis technology.

[1176] input

[1177] Response Text

[1178] output

[1179] Audio data

[1180] Specific actions

[1181] The server calls a speech synthesis API (e.g., Amazon Polly) to convert the text to speech.

[1182] Step 7: Sending a response audio

[1183] The server transmits the generated voice data to the user's terminal.

[1184] input

[1185] Audio data

[1186] output

[1187] Sending audio to the user

[1188] Specific actions

[1189] The server sends the audio data to the terminal as an HTTP response.

[1190] Step 8: Play the audio response

[1191] The terminal plays this audio data and provides a response to the user.

[1192] input

[1193] Audio data

[1194] output

[1195] Playing audio

[1196] Specific actions

[1197] The device plays the audio data using a playback application.

[1198] Feedback System

[1199] Step 1: Sending a termination notice

[1200] After the meeting is over, the user sends a notification of the end to the system.

[1201] input

[1202] Termination Notice

[1203] output

[1204] Sending notifications to the server

[1205] Specific actions

[1206] The user clicks the end meeting button and a notification is sent to the server.

[1207] Step 2: Receive notifications

[1208] The terminal sends a termination notification to the server.

[1209] input

[1210] Termination Notice

[1211] output

[1212] Sending notifications to the server

[1213] Specific actions

[1214] The terminal sends a termination notification to the server as an HTTP request.

[1215] Step 3: Analyzing session content

[1216] The server analyzes the session content based on the received notification and evaluates the user's learning progress.

[1217] input

[1218] End notice and session details

[1219] output

[1220] Evaluation Data

[1221] Specific actions

[1222] The server analyzes the session data using machine learning algorithms.

[1223] Step 4: Create a feedback report

[1224] The server generates a feedback report based on the analysis results.

[1225] input

[1226] Evaluation Data

[1227] output

[1228] Feedback Report

[1229] Specific actions

[1230] The server runs an algorithm that automatically generates a feedback report.

[1231] Step 5: Submit the report

[1232] The server sends the generated feedback report to the user.

[1233] input

[1234] Feedback Report

[1235] output

[1236] Sending reports to users

[1237] Specific actions

[1238] The server sends the feedback report as an HTTP response.

[1239] Step 6: Review the report

[1240] Users can check the feedback report and use it to plan their future studies.

[1241] input

[1242] Feedback Report

[1243] output

[1244] Verified Feedback

[1245] Specific actions

[1246] The user displays the feedback report on the device and checks the contents.

[1247] (Application example 1)

[1248] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1249] Current factory operations often rely on human resources for troubleshooting and work guidance, which can lead to reduced productivity and the need for rapid response when problems occur. Furthermore, it can take time for staff to learn new technologies and operating procedures, making it difficult for them to respond immediately. Therefore, there is a need for a system that can provide real-time problem solving and guidance through a voice interface.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1251] In this invention, the server includes a means for recognizing user voice data and converting it into text data, a means for inputting the text data into a generating artificial intelligence, which then generates a response, and a means for providing the generated voice response to the user, thereby enabling quick troubleshooting and work guidance within the factory.

[1252] "User" refers to the person or factory staff who uses the system.

[1253] An "expert" is a person or system with advanced specialized knowledge or skills.

[1254] A "session" is a period of time during which information is exchanged or consulted between a user and an expert.

[1255] A "reservation" is a prior application or registration to use a system or expert at a specific date and time.

[1256] A "server" is a computer system that processes reservation information and voice data and inputs them into the generation AI.

[1257] "Voice data" refers to data in which a user's questions or requests are recorded in the form of sound.

[1258] "Text data" refers to data obtained by converting voice data into character information.

[1259] "Generative artificial intelligence" refers to an AI system that generates appropriate responses based on input text data.

[1260] "Response text" refers to data that expresses the answer content generated by the generating AI in text.

[1261] A "feedback report" is a report that summarizes the user's learning progress and areas for improvement after a session.

[1262] System program description

[1263] The system for realizing this invention is constructed using the following hardware and software: a user's terminal, a server, a generative AI model, speech recognition technology, and speech synthesis technology.

[1264] Hardware and Software Use

[1265] 1. On the user's device:

[1266] Microphone (audio input)

[1267] Speaker (audio output)

[1268] Internet connection function (data communication)

[1269] 2. Server:

[1270] Database (storing reservation information)

[1271] Speech recognition technology (converts voice to text)

[1272] Generative AI models (generate responses from text)

[1273] Speech synthesis technology (converting text to speech)

[1274] Details of data processing and calculation

[1275] The server receives the user's voice data and converts it into text using speech recognition technology, a process that uses software such as speech_recognition.

[1276] The converted text data is then fed into a generative artificial intelligence system, which uses a generative AI model such as GPT-2 to generate an optimal response based on the input text.

[1277] The generated text response is converted into voice data using speech synthesis technology, using software such as pyttsx3.

[1278] Finally, this voice data is sent to the user's terminal and presented to the user through a speaker.

[1279] Specific examples

[1280] Reservation System:

[1281] Example: When a user books a session with an expert, they enter their information and desired date and time in the terminal interface. The information is sent to the server and stored in a database. The server generates a unique booking ID and sends it back to the user.

[1282] session:

[1283] Example: On the day of the session, the user voice-inputs the question, "What should I do to learn a new programming language?" The server converts the voice data into text and feeds it into a generative AI model. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This text response is converted into speech and provided to the user's device.

[1284] Prompt Sentence Examples

[1285] Here is an example prompt for the example above:

[1286] Generate an appropriate response to "What caused this machine in the factory to stop working?"

[1287] This will enable quick troubleshooting and work guidance within the factory, and is expected to improve productivity.

[1288] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1289] Step 1:

[1290] The user opens the reservation interface on the terminal to reserve a session with an expert. The user enters their information (name and ID), the desired expert, and the date and time of the session. The terminal sends the reservation information entered by the user to the server. The server receives the reservation information and stores it in a database. The server generates a unique reservation ID and returns it to the user via the terminal.

[1291] Step 2:

[1292] On the day of the session, the user uses the microphone on the device to ask the expert a question. The device records the user's voice data and sends it to the server. The input at this time is the user's voice data.

[1293] Step 3:

[1294] The server converts the received voice data into text data using speech recognition software. This process uses the speech_recognition software. By converting the voice data into text, the content of the user's question is output as text data.

[1295] Step 4:

[1296] The server inputs the user's question, converted into text, into a generative AI model. GPT-2 is used as the generative AI model. The text data is passed as input to the AI ​​model, which generates an appropriate response text. The generative AI model generates a response using a prompt sentence based on the user's question. Specifically, a response based on the prompt sentence, "Please generate an appropriate reply to the question 'What caused this machine in the factory to stop?'" is output.

[1297] Step 5:

[1298] The generated response text is converted into voice data by the speech synthesis system in the server. For this purpose, we use the pyttsx3 software, which generates voice data from the text data and outputs the response voice.

[1299] Step 6:

[1300] The server transmits the generated voice data to the terminal, which receives the voice data and provides a response to the user through a speaker, where the output is a voice response that the user can hear.

[1301] Step 7:

[1302] After the session ends, the user sends a notification to the server via their device. The server receives this notification, analyzes the session, and creates a feedback report. This analysis and feedback includes usage data, user questions, and AI responses.

[1303] Step 8:

[1304] The server sends the created feedback report to the terminal, which displays the report to the user, allowing the user to use it for future learning and work improvement. The output at this time is the feedback report.

[1305] The above process realizes a system that allows users to quickly receive expert guidance and troubleshooting suggestions in real time using voice.

[1306] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1307] Based on the problem to be solved by the present invention, a specific embodiment for carrying out the invention in combination with an emotion engine will be described below.

[1308] 1. Reservation System Implementation

[1309] The user opens an interface on their terminal to reserve a session with an expert and enters the necessary reservation information, such as their ID, desired expert, and desired session date and time. The terminal sends this reservation information to the server's reservation API via a POST request. The server receives the reservation information and generates a unique reservation ID when saving it in the database. The server then notifies the user of the generated reservation ID.

[1310] 2. Session Format

[1311] At the start of a session, the user inputs a question or task by voice. The device records the user's voice and sends the voice data to the server. The server analyzes the received voice data, converts it into text using voice recognition technology, and then recognizes the user's emotions using an emotion engine. Based on the recognized emotions, the dialogue AI system generates an appropriate response. The generated text response is converted back into voice by the server and sent to the device. The device plays back this voice data and provides the response to the user.

[1312] 3. Feedback System Embodiments

[1313] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[1314] Specific examples using emotion engines

[1315] Consider a scenario where, during a session, the user asks a question by voice, such as, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes the text data to the emotion engine. The emotion engine analyzes the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, the response generated might be, "Start with a basic tutorial, then hone your skills with practical projects." The server then converts this text back into audio data and provides it to the user.

[1316] Dynamic Adjustment Examples

[1317] The server evaluates the user's emotions in real time during the session and dynamically adjusts the expert's instruction. For example, if the user shows signs of anxiety, the expert will flexibly change the instruction to suit the situation. This allows the user to learn in a relaxed state and effectively acquire skills.

[1318] Example of a feedback report

[1319] After the session ends, the server receives a notification and analyzes the session content. This analysis includes the user's voice data, text data, and emotional history. Based on the analysis results, the server provides feedback such as, "The user has grasped basic knowledge, and in the future, more practical projects will further improve their skills." The feedback report is sent to the user, who can use it to plan their next learning.

[1320] In this way, a system can be implemented in which users can smoothly progress through learning by utilizing the emotion engine while receiving guidance from an expert, and can efficiently improve their skills.

[1321] The processing flow will be explained below.

[1322] Step 1:

[1323] The user opens an interface on the terminal for reserving a session with an expert and inputs necessary reservation information such as the user's ID, the desired expert, and the desired date and time of the session.

[1324] Step 2:

[1325] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[1326] Step 3:

[1327] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[1328] Step 4:

[1329] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[1330] Step 5:

[1331] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[1332] Step 6:

[1333] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[1334] Step 7:

[1335] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[1336] Step 8:

[1337] The server analyzes the received voice data and converts the voice data into text using speech recognition, which is then input to the emotion engine.

[1338] Step 9:

[1339] The server uses an emotion engine to recognize the user's emotions from their voice, and classifies the emotions into categories such as "happiness," "sadness," "anger," and "surprise."

[1340] Step 10:

[1341] The server inputs text data based on the user's emotions into the conversational AI system, which then generates an appropriate response. For example, if the user expresses "anxiety," the AI ​​will generate a more friendly and reassuring response.

[1342] Step 11:

[1343] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[1344] Step 12:

[1345] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[1346] Step 13:

[1347] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[1348] Step 14:

[1349] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio data, text data, and emotion history collected during the session.

[1350] Step 15:

[1351] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[1352] Step 16:

[1353] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[1354] Step 17:

[1355] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[1356] Step 18:

[1357] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[1358] Example 2

[1359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1360] Conventional reservation systems have the drawback of requiring users to perform many manual operations when reserving a session with an expert, and making it difficult to manage reservation information. Furthermore, they are unable to grasp the user's emotions in real time during the session and dynamically adjust the expert's instruction content based on those emotions, resulting in insufficient learning effectiveness. Furthermore, feedback after the session is sometimes delayed, making it difficult to adequately support the user's learning progress.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1362] In this invention, the server includes means for transmitting the user's reservation information to the server and saving the reservation information in a database, means for generating a specific reservation number and providing it to the user, means for transmitting voice data from the user to the server, means for converting the voice data to text, means for inputting the text data into an emotion analysis engine and recognizing emotions, means for generating a response using a generative AI model based on the recognized emotions, means for converting the generated response text into voice and providing it to the user, means for receiving a session end notification from the user and analyzing the session content and the user's emotion history to make a final evaluation, and means for creating a feedback report and sending it to the user. This enables the user to make a reservation smoothly, receive appropriate responses in real time during the session, and receive prompt feedback after the session ends.

[1363] A "user" is a person who utilizes the system to book and conduct sessions with experts.

[1364] An "expert" is someone who has extensive knowledge and experience in a particular field and who can provide guidance and advice to users.

[1365] A "server" is a computer system that receives information from users, processes the data, stores it in a database, and runs generative AI models.

[1366] A "terminal" is a device used by a user to access the system, and includes a PC, smartphone, tablet, etc.

[1367] A "generative AI model" is an artificial intelligence algorithm that generates optimal responses or predictions based on specific input data.

[1368] A "reservation number" is a unique identifier that allows a user to identify a reservation.

[1369] "Voice data" refers to data in which the user's voice is recorded in digital format.

[1370] "Text data" refers to character data obtained by analyzing and transcribing voice data.

[1371] An "emotion analysis engine" is an analysis device or program that analyzes a user's emotions (e.g., excitement, anxiety, joy, etc.) from input text data.

[1372] A "feedback report" is a report that includes an evaluation and suggestions for future learning, generated based on the user's performance and emotional history during the session.

[1373] This invention is a system that allows users to reserve sessions with experts, and receive feedback on the sessions. This system is composed of three entities: a server, a terminal, and a user, and operates as follows.

[1374] 1. Reservation System Implementation

[1375] To book a session with an expert, a user opens the device's interface and enters the necessary information, such as their ID, the preferred expert, and the desired date and time of the session. This interface is provided as a web page or mobile app. The device sends this reservation information to the server's reservation API via an HTTP POST request. The server executes an SQL query to store the received reservation information in a database and generates a unique reservation number. The server then returns a response including the generated reservation number to the device, and the user is notified. This allows the user to confirm that the reservation was successful.

[1376] 2. Session Format

[1377] At the start of a session, the user uses the voice input function to speak a question or task into the device. The device records the user's voice and sends the audio data to the server. This audio data is saved in an appropriate format (such as .wav or .mp3). The server analyzes the received audio data and converts it into text using speech recognition technology (e.g., Google Cloud Speech-to-Text). The server then passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. Based on the recognized emotion, a generative AI model (e.g., OpenAI GPT-3) generates a response. This response is again converted into speech by the server (e.g., Amazon Polly) and sent to the device. The device plays the audio data and provides the response to the user.

[1378] 3. Feedback System Embodiments

[1379] After the session ends, the user sends a completion notification to the system. The device then sends this completion notification to the server. The server analyzes the session content based on the received completion notification (e.g., using an NLP analysis tool) and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning.

[1380] Add specific examples to the description

[1381] Reservation system example

[1382] For example, if a user wants to reserve a session with an expert "Expert A" with ID "user123" for 15:00 on November 1, 2023, he or she enters this information into the terminal interface. The terminal sends this information to the server, which stores it in its database, generates a reservation number "reserve456" and notifies the user.

[1383] Session Example

[1384] For example, if a user asks, "What should I do to learn a new programming language?", the device records the question and sends it to a server. The server converts the speech to text using Google Cloud Speech-to-Text and analyzes sentiment using IBM Watson Tone Analyzer. OpenAI GPT-3 generates a response such as "Start with basic tutorials and hone your skills with hands-on projects," and Amazon Polly converts this text to speech and serves it to the user.

[1385] Examples of feedback systems

[1386] After the session ends, the server receives a notification from the user and analyzes the user's emotional history and session data. It then generates feedback to the user, such as "The user has grasped basic knowledge and would benefit from more practical projects."

[1387] Prompt Sentence Examples

[1388] "What should I do when learning a new programming language?"

[1389] Through each of these steps, users can enjoy a smooth booking, session, and feedback experience, and a system can be implemented that allows them to efficiently improve their skills while receiving expert guidance.

[1390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1391] Step 1:

[1392] The user opens the device interface and enters information such as their ID, the preferred specialist, and the desired date and time of the session. This information is entered into the device as reservation information. The device temporarily stores this information and displays a confirmation screen to the user.

[1393] Step 2:

[1394] When the user confirms the reservation information and presses the send button, the device sends this reservation information to the server's reservation API via an HTTP POST request. The input data is the user ID, expert ID, and reservation date and time, and is sent to the server in JSON format.

[1395] Step 3:

[1396] The server analyzes the reservation information received from the terminal and generates an SQL query to store it in the database. Here, the user ID, expert ID, and reservation date and time are input data, and are stored in fields in the "Reservation" table. The server generates a unique reservation number and returns it as a response.

[1397] Step 4:

[1398] The terminal receives the response from the server and displays a confirmation message to the user, including the reservation number. This confirms that the reservation was successfully completed. The output is the reservation number and the confirmation message.

[1399] Step 5:

[1400] At the scheduled time, the user starts the session using the device. Using the voice input function, the user voice-enters questions and tasks into the device. This voice data becomes the input data.

[1401] Step 6:

[1402] The device records the user's voice and sends it to the server. The voice data is encoded in the appropriate format (e.g., .wav or .mp3). The data sent to the server is the binary data of the voice file.

[1403] Step 7:

[1404] The server analyzes the received voice data and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The input data is an audio file, and the output is text data. The server temporarily stores the text data.

[1405] Step 8:

[1406] The server passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data, and the output is the emotion recognition result (e.g., excitement, anxiety, etc.). The server temporarily stores the emotion recognition result.

[1407] Step 9:

[1408] The server inputs the recognized emotion and text data into a generative AI model (e.g., OpenAI GPT-3) to generate an appropriate response. The input data is text and emotion recognition results, and the output is a response text. The server temporarily stores the generated response text.

[1409] Step 10:

[1410] The server converts the generated response text into speech using speech synthesis technology (e.g., Amazon Polly). The input data is the response text, and the output is an audio file. The server then sends the audio data to the device.

[1411] Step 11:

[1412] The terminal plays back the voice data received from the server and provides a response to the user. The output is a voice response, allowing the user to experience a dialogue in real time.

[1413] Step 12:

[1414] After the session ends, the user sends a termination notification to the system. By clicking the end button, the terminal sends a termination notification to the server. The data sent is a termination notification flag.

[1415] Step 13:

[1416] The server receives the termination notification and analyzes the audio data, text data, and emotion history during the session (e.g., using an NLP analysis tool). The input data are audio data, text data, and emotion recognition results, and the output is the analysis results.

[1417] Step 14:

[1418] The server generates a feedback report based on the analysis results. The input data is the analysis results, and the output is the feedback report. The report contains information that is useful for the user's learning progress and future learning plans.

[1419] Step 15:

[1420] The server sends a feedback report to the user, and the sent data is the feedback report, which the user can check and use for future learning.

[1421] (Application example 2)

[1422] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1423] Conventional online session systems lack support for users to communicate effectively with experts. Furthermore, because they respond uniformly without considering the user's emotions, it is difficult to provide appropriate support for emotions such as anxiety and excitement. Furthermore, because feedback does not take into account the user's emotions or learning progress, it is difficult to create an effective learning plan.

[1424] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing user reservation information in a database, means for analyzing the user's voice input in real time based on the generated reservation information and recognizing emotions, means for generating an appropriate response to the voice input based on the emotion recognition, and means for dynamically adjusting and reproducing the response taking the user's emotions into consideration. This makes it possible to grasp the user's emotions in real time and provide an optimal response. In addition, by evaluating the user's learning progress and providing appropriate feedback, it is possible to create an effective learning plan.

[1425] "User" refers to a person who uses the system to hold a session with an expert.

[1426] An "expert" refers to a person or system that has specialized knowledge and skills in a particular field and that holds a session with a user to provide assistance.

[1427] A "session" refers to a period of interaction or instruction between a user and an expert for a specific purpose.

[1428] "Reservation information" refers to information provided by a user when reserving a session with an expert, and includes, for example, an ID, a desired expert, and a desired date and time for the session.

[1429] "Server" refers to a central processing unit that processes data and communicates between users and experts.

[1430] "Database" refers to a system that centrally manages and stores data such as reservation information.

[1431] A "reservation ID" is a unique identifier generated by the server and used to manage session reservations.

[1432] "Analyzing in real time" refers to a process of instantly analyzing voice data while the user is inputting voice data.

[1433] "Emotion recognition" refers to analyzing and identifying emotional states such as excitement, anxiety, and joy from a user's voice and text data.

[1434] "Dynamic response adjustment" refers to changing and optimizing expert guidance and system responses in real time according to the user's emotional state.

[1435] A "feedback report" is a report generated by the server after a session ends. It evaluates the user's learning progress and emotional history, and provides information that can be used as a reference for future learning plans.

[1436] The present invention provides a product purchasing support system in a virtual store that allows users to reserve a session with an expert and dynamically responds to user emotions. Specific embodiments of this system are described below.

[1437] 1. Reservation System Implementation

[1438] A user uses their device (e.g., smartphone, tablet, or computer) to reserve a session with an expert. The user opens the reservation interface and enters the necessary information, such as their ID, the desired expert, and the desired session date and time. The device then sends this reservation information to the server's reservation API via a POST request. The server stores the received reservation information in a database and generates a unique reservation ID. This reservation ID is notified to the user.

[1439] 2. Session Format

[1440] At the start of a session, the user vocally inputs a question or task related to the session content. The device records the user's voice and sends the voice data to the server. The server then uses speech recognition technology (e.g., Google Speech Recognition API) to convert the voice data into text. It then uses an emotion recognition engine (e.g., EmotionRecognizer library) to analyze and recognize the user's emotions from the text data. Based on the recognized emotions, the server's conversational AI system (e.g., generative AI model) generates an appropriate response. The generated text response is converted into voice data by the server (e.g., TextToSpeech library), sent to the device, and provided to the user.

[1441] 3. Feedback System Embodiments

[1442] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning. In this way, the user can effectively progress with their learning by utilizing the emotion engine while receiving guidance from an expert.

[1443] Specific examples

[1444] For example, consider a case where a user asks by voice, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes this text data to an emotion recognition engine. The emotion recognition engine analyzes emotions from the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, it might generate a response as text, such as "Start with a basic tutorial and then hone your skills with practical projects," and the server then converts this text back into audio data and provides it to the user.

[1445] Prompt Sentence Examples

[1446] User Question: "What should I do when learning a new programming language?"

[1447] Recognized emotion: "excitement"

[1448] This allows users to receive expert guidance in a way that suits their emotions, resulting in a more effective and personalized purchasing experience and learning.

[1449] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1450] Step 1:

[1451] The user opens the reservation interface and enters reservation information such as ID, desired expert, desired session date and time, etc. The terminal sends this input information to the server. The server stores the received reservation information in a database and generates a unique reservation ID. The server notifies the user of this reservation ID.

[1452] Step 2:

[1453] At the start of a session, the user enters a question or task by voice through the device. The device records the user's voice and sends the voice data to the server. The server receives the voice data and converts it into text using voice recognition technology.

[1454] Step 3:

[1455] The server inputs the converted text data into an emotion recognition engine, which analyzes the text data and identifies the user's emotion. The emotion recognition engine outputs the user's emotional state (e.g., excitement, anxiety).

[1456] Step 4:

[1457] The server generates an appropriate response using a generative AI model based on the recognized emotion. The generative AI model creates an optimal response text based on the user's question content and emotional state. The generative AI model outputs this response text.

[1458] Step 5:

[1459] The server uses the TextToSpeech library to convert the response text into audio data, which is then sent to the device, which plays the audio response, and the user receives the response as audio.

[1460] Step 6:

[1461] After the session ends, the user sends a notification to the server. The server receives the notification and analyzes the session contents, including the user's voice data, text data, and emotional history.

[1462] Step 7:

[1463] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's learning progress and emotional state. The server then sends this feedback report to the terminal and provides it to the user.

[1464] The above are the specific processing steps of the system for realizing the present invention.

[1465] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1466] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1467] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1468] [Fourth embodiment]

[1469] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1470] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1471] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1472] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1473] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1474] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1475] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1476] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1477] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1480] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1481] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1482] Based on the problems to be solved by the present invention, specific embodiments for carrying out the invention will be described below.

[1483] 1. Reservation System Implementation

[1484] The user opens an interface on the terminal to reserve a session with an expert. In this interface, the user enters their ID, the desired expert, and the date and time of the session. The terminal sends this reservation information to the server. The server receives the reservation information and stores it in a database. The server then generates a unique reservation ID and returns it to the user.

[1485] 2. Session Format

[1486] On the day of the session, the user asks the expert a question by voice in order to participate in the session. The terminal records the user's voice and sends the voice data to the server. The server analyzes the received voice data and converts it into text using voice recognition technology. The converted text data is input into a generative AI system. The generative AI system generates an appropriate response to the user's question. This response text is again converted into voice by the server and sent to the user's terminal. The terminal plays back the voice data and provides the response to the user.

[1487] 3. Feedback System Embodiments

[1488] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress. It creates a feedback report based on the analysis results and sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[1489] Specific examples

[1490] Consider a specific example where a user wants to book a session with an expert. If User A wants to book a session with Expert B at 3:00 PM on November 1st, User A enters his / her information and desired date and time into the booking interface. The terminal sends this to the server, which stores the booking information in a database, generates a booking ID 12345, and notifies User A.

[1491] On the day of the session, User A voice-inputs the question, "What should I do to learn a new programming language?" The device records the voice and sends it to the server. The server converts the voice data into text and inputs it into the generative AI. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with practical projects." This text response is converted back into speech and provided to User A.

[1492] When the session ends, User A sends a completion notification, and the server analyzes the session content and creates a feedback report. The feedback report includes an evaluation such as "User A understands basic knowledge, but more practical projects would further improve his skills." The user can use this feedback to plan his next learning experience.

[1493] In this way, a system can be implemented that allows users to efficiently improve their skills while receiving guidance from experts.

[1494] The processing flow will be explained below.

[1495] Step 1:

[1496] The user opens an interface on the terminal to reserve a session with an expert, and enters the necessary reservation information, such as the user's ID, the desired expert, and the desired session date and time.

[1497] Step 2:

[1498] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[1499] Step 3:

[1500] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[1501] Step 4:

[1502] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[1503] Step 5:

[1504] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[1505] Step 6:

[1506] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[1507] Step 7:

[1508] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[1509] Step 8:

[1510] The server analyzes the received voice data and converts it into text using speech recognition, which is then input into the conversational AI system.

[1511] Step 9:

[1512] The server uses a conversational AI system to generate appropriate responses to user questions and challenges, and the generated text responses are stored.

[1513] Step 10:

[1514] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[1515] Step 11:

[1516] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[1517] Step 12:

[1518] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[1519] Step 13:

[1520] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio and text data collected during the session.

[1521] Step 14:

[1522] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[1523] Step 15:

[1524] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[1525] Step 16:

[1526] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[1527] Step 17:

[1528] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[1529] Example 1

[1530] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1531] Conventional systems for booking meetings with experts require a complicated process from reservation to actual meeting, and lack the technology to make it more efficient. Furthermore, evaluating users' learning progress and providing feedback after the meeting is time-consuming, which reduces the overall quality of the service. Another problem is the lack of an established process for efficiently converting voice input into text and generating appropriate responses using artificial intelligence.

[1532] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1533] In this invention, the server includes: means for a user to reserve a meeting with an expert; means for transmitting the user's reservation information to a communication device and for the communication device to store the reservation information in a storage device; means for providing the user with a generated reservation identifier; means for the communication device to convert voice data into text; means for inputting the generated text data into an artificial intelligence system and for the artificial intelligence system to generate a response; and means for converting the generated response text into voice and providing it to the user. This enables a consistent and efficient process from reserving a meeting to holding it and providing feedback after it has ended.

[1534] "Users" are those who use the system to schedule meetings with experts and ask questions or provide feedback.

[1535] An "expert" is a person from whom a user can meet and receive knowledge and advice.

[1536] A "meeting" is a session in which users and experts interact and ask questions.

[1537] "Reservation information" refers to information that a user enters for a meeting, such as their ID, the desired expert, and the date and time of the meeting.

[1538] A "communication device" is an electronic device that transmits user input data and voice data to a server and conveys the user's response.

[1539] A "storage device" is a storage system that includes a database in which user reservation information and the like are stored.

[1540] A "Reservation Identifier" is a generated unique ID that identifies a particular reservation.

[1541] "Voice data" is digital data of the voice input by the user to ask a question to an expert.

[1542] "Text data" is character information converted from voice data using voice recognition technology.

[1543] An "artificial intelligence system" is a machine learning-based system that generates appropriate responses based on a user's text data.

[1544] "Response text" is a textual response to a user's question generated by an artificial intelligence system.

[1545] "Speech synthesis technology" is a technology for converting text data into voice data.

[1546] An "evaluation report" is a document summarizing the results of analyzing the session content and evaluating the user's learning progress.

[1547] MODE FOR CARRYING OUT THE INVENTION

[1548] The system of the present invention is designed to efficiently and automatically carry out the entire process of a user reserving a meeting with an expert, conducting the meeting, and receiving feedback after the meeting. This system operates mainly through three parties: a server, a terminal, and a user.

[1549] Reservation System Embodiment

[1550] Users can book a meeting with an expert using a dedicated interface. Specifically, users enter their ID, the desired expert, and the date and time of the meeting. This interface is built using libraries such as React and Angular on the front end, Node.js and Django on the back end, and MySQL and PostgreSQL as the database.

[1551] The terminal sends the collected reservation information to the server, which stores the received reservation information in a database, generates a unique reservation identifier (reservation ID), and returns it to the user.

[1552] For example, the user enters "userA, expert B, 2023-11-01 15:00" and the terminal sends the reservation information to the server. The server saves the reservation information in a storage device, generates a reservation ID "12345", and returns it to the user.

[1553] Meeting System Embodiments

[1554] On the day of the meeting, the user speaks their question to the expert. The device records the user's voice data and sends it to the server, which then converts the voice data into text using a speech recognition service (such as Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[1555] The converted text data is then fed into a generative AI model (such as OpenAI GPT-4 or Google BERT) to generate an appropriate response. The generated response text is then converted into audio data using speech synthesis technology and sent to the user's device. The device then plays back the audio data and provides it to the user.

[1556] For example, a user may ask, "What should I do to learn a new programming language?" The server converts the voice data into text, and the generative AI generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This response is then converted into audio and provided to the user.

[1557] Feedback System Embodiments

[1558] After the meeting, the user sends a notification to the system. The device then sends this notification to the server, which then analyzes the session content and evaluates the user's learning progress. Machine learning techniques can be used for the analysis.

[1559] The server then creates a feedback report based on the evaluation results and sends it to the user, who can review the feedback report and use it to plan their next study.

[1560] As a concrete example, a user sends a completion notification to the server, and the server generates a feedback report stating, "User A understands basic knowledge, but his skills will improve further with more practical projects," and sends it to the user.

[1561] Prompt Sentence Examples

[1562] Please provide a detailed explanation, using examples, of how a user schedules a meeting with an expert, conducts the meeting, and receives feedback. Please also include specific actions taken by the server, device, and user during this process.

[1563] This system allows users to efficiently go through a series of processes while receiving expert guidance.

[1564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1565] Specific explanation of processing steps

[1566] Reservation System

[1567] Step 1: View the interface

[1568] The terminal displays an interface for the user to schedule a meeting with an expert, the interface including fields for entering the user ID, the desired expert, and the date and time of the meeting.

[1569] input

[1570] User operation (interface display request)

[1571] output

[1572] View the booking interface

[1573] Specific actions

[1574] The terminal loads the reservation interface HTML page and displays it to the user.

[1575] Step 2: Enter reservation information

[1576] The user enters their ID, the desired expert, and the date and time of the meeting into the interface.

[1577] input

[1578] User input information (ID, expert, date and time)

[1579] output

[1580] Reservation information entered

[1581] Specific actions

[1582] The user enters information using a keyboard or touch screen.

[1583] Step 3: Submit your reservation information

[1584] The terminal transmits the reservation information input by the user to the server.

[1585] input

[1586] Reservation information (ID, specialist, date and time)

[1587] output

[1588] Sending data to the server

[1589] Specific actions

[1590] The terminal sends the reservation information to the server as an HTTP request.

[1591] Step 4: Save your reservation

[1592] The server stores the received reservation information in a database.

[1593] input

[1594] Received reservation information

[1595] output

[1596] Reservation information recorded in the database

[1597] Specific actions

[1598] The server converts the reservation information into the appropriate format and stores it in a database using SQL.

[1599] Step 5: Generate a reservation ID

[1600] The server generates a unique reservation identifier (reservation ID).

[1601] input

[1602] Saved reservations

[1603] output

[1604] Generated reservation ID

[1605] Specific actions

[1606] The server runs an algorithm to generate a unique ID based on the reservation information.

[1607] Step 6: Reservation ID response

[1608] The server returns the generated reservation ID to the user.

[1609] input

[1610] Generated reservation ID

[1611] output

[1612] Sending the reservation ID to the user

[1613] Specific actions

[1614] The server creates an HTTP response including the reservation ID and sends it to the terminal.

[1615] Meeting System

[1616] Step 1: Enter your voice question

[1617] On the day of the session, the user will use the terminal to voice-type questions to the expert.

[1618] input

[1619] User voice questions

[1620] output

[1621] Audio data

[1622] Specific actions

[1623] The user speaks a question into the microphone.

[1624] Step 2: Sending audio data

[1625] The terminal records the user's voice and transmits the voice data to the server.

[1626] input

[1627] Audio data

[1628] output

[1629] Sending audio data to the server

[1630] Specific actions

[1631] The device records the audio and sends it to the server in WAV or MP3 format.

[1632] Step 3: Convert audio data to text

[1633] The server converts the received voice data into text using voice recognition technology.

[1634] input

[1635] Audio data

[1636] output

[1637] Text data

[1638] Specific actions

[1639] The server calls a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the speech into text.

[1640] Step 4: AI input of text data

[1641] The server inputs the converted text data into a generative AI model.

[1642] input

[1643] Text data

[1644] output

[1645] Input data for AI models

[1646] Specific actions

[1647] The server converts the text data into a format suitable for the AI ​​model and sends it to the model.

[1648] Step 5: Generate an AI response

[1649] Generative AI models (e.g., OpenAI GPT-4) generate appropriate responses to user questions.

[1650] input

[1651] Input data for AI models

[1652] output

[1653] Response Text

[1654] Specific actions

[1655] The AI ​​model performs the calculations to generate an answer to the question.

[1656] Step 6: Convert response text to speech

[1657] The server converts the generated response text into voice data using voice synthesis technology.

[1658] input

[1659] Response Text

[1660] output

[1661] Audio data

[1662] Specific actions

[1663] The server calls a speech synthesis API (e.g., Amazon Polly) to convert the text to speech.

[1664] Step 7: Sending a response audio

[1665] The server transmits the generated voice data to the user's terminal.

[1666] input

[1667] Audio data

[1668] output

[1669] Sending audio to the user

[1670] Specific actions

[1671] The server sends the audio data to the terminal as an HTTP response.

[1672] Step 8: Play the audio response

[1673] The terminal plays this audio data and provides a response to the user.

[1674] input

[1675] Audio data

[1676] output

[1677] Playing audio

[1678] Specific actions

[1679] The device plays the audio data using a playback application.

[1680] Feedback System

[1681] Step 1: Sending a termination notice

[1682] After the meeting is over, the user sends a notification of the end to the system.

[1683] input

[1684] Termination Notice

[1685] output

[1686] Sending notifications to the server

[1687] Specific actions

[1688] The user clicks the end meeting button and a notification is sent to the server.

[1689] Step 2: Receive notifications

[1690] The terminal sends a termination notification to the server.

[1691] input

[1692] Termination Notice

[1693] output

[1694] Sending notifications to the server

[1695] Specific actions

[1696] The terminal sends a termination notification to the server as an HTTP request.

[1697] Step 3: Analyzing session content

[1698] The server analyzes the session content based on the received notification and evaluates the user's learning progress.

[1699] input

[1700] End notice and session details

[1701] output

[1702] Evaluation Data

[1703] Specific actions

[1704] The server analyzes the session data using machine learning algorithms.

[1705] Step 4: Create a feedback report

[1706] The server generates a feedback report based on the analysis results.

[1707] input

[1708] Evaluation Data

[1709] output

[1710] Feedback Report

[1711] Specific actions

[1712] The server runs an algorithm that automatically generates a feedback report.

[1713] Step 5: Submit the report

[1714] The server sends the generated feedback report to the user.

[1715] input

[1716] Feedback Report

[1717] output

[1718] Sending reports to users

[1719] Specific actions

[1720] The server sends the feedback report as an HTTP response.

[1721] Step 6: Review the report

[1722] Users can check the feedback report and use it to plan their future studies.

[1723] input

[1724] Feedback Report

[1725] output

[1726] Verified Feedback

[1727] Specific actions

[1728] The user displays the feedback report on the device and checks the contents.

[1729] (Application example 1)

[1730] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1731] Current factory operations often rely on human resources for troubleshooting and work guidance, which can lead to reduced productivity and the need for rapid response when problems occur. Furthermore, it can take time for staff to learn new technologies and operating procedures, making it difficult for them to respond immediately. Therefore, there is a need for a system that can provide real-time problem solving and guidance through a voice interface.

[1732] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1733] In this invention, the server includes a means for recognizing user voice data and converting it into text data, a means for inputting the text data into a generating artificial intelligence, which then generates a response, and a means for providing the generated voice response to the user, thereby enabling quick troubleshooting and work guidance within the factory.

[1734] "User" refers to the person or factory staff who uses the system.

[1735] An "expert" is a person or system with advanced specialized knowledge or skills.

[1736] A "session" is a period of time during which information is exchanged or consulted between a user and an expert.

[1737] A "reservation" is a prior application or registration to use a system or expert at a specific date and time.

[1738] A "server" is a computer system that processes reservation information and voice data and inputs them into the generation AI.

[1739] "Voice data" refers to data in which a user's questions or requests are recorded in the form of sound.

[1740] "Text data" refers to data obtained by converting voice data into character information.

[1741] "Generative artificial intelligence" refers to an AI system that generates appropriate responses based on input text data.

[1742] "Response text" refers to data that expresses the answer content generated by the generating AI in text.

[1743] A "feedback report" is a report that summarizes the user's learning progress and areas for improvement after a session.

[1744] System program description

[1745] The system for realizing this invention is constructed using the following hardware and software: a user's terminal, a server, a generative AI model, speech recognition technology, and speech synthesis technology.

[1746] Hardware and Software Use

[1747] 1. On the user's device:

[1748] Microphone (audio input)

[1749] Speaker (audio output)

[1750] Internet connection function (data communication)

[1751] 2. Server:

[1752] Database (storing reservation information)

[1753] Speech recognition technology (converts voice to text)

[1754] Generative AI models (generate responses from text)

[1755] Speech synthesis technology (converting text to speech)

[1756] Details of data processing and calculation

[1757] The server receives the user's voice data and converts it into text using speech recognition technology, a process that uses software such as speech_recognition.

[1758] The converted text data is then fed into a generative artificial intelligence system, which uses a generative AI model such as GPT-2 to generate an optimal response based on the input text.

[1759] The generated text response is converted into voice data using speech synthesis technology, using software such as pyttsx3.

[1760] Finally, this voice data is sent to the user's terminal and presented to the user through a speaker.

[1761] Specific examples

[1762] Reservation System:

[1763] Example: When a user books a session with an expert, they enter their information and desired date and time in the terminal interface. The information is sent to the server and stored in a database. The server generates a unique booking ID and sends it back to the user.

[1764] session:

[1765] Example: On the day of the session, the user voice-inputs the question, "What should I do to learn a new programming language?" The server converts the voice data into text and feeds it into a generative AI model. The AI ​​generates a response such as, "Start with basic tutorials and hone your skills with hands-on projects." This text response is converted into speech and provided to the user's device.

[1766] Prompt Sentence Examples

[1767] Here is an example prompt for the example above:

[1768] Generate an appropriate response to "What caused this machine in the factory to stop working?"

[1769] This will enable quick troubleshooting and work guidance within the factory, and is expected to improve productivity.

[1770] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1771] Step 1:

[1772] The user opens the reservation interface on the terminal to reserve a session with an expert. The user enters their information (name and ID), the desired expert, and the date and time of the session. The terminal sends the reservation information entered by the user to the server. The server receives the reservation information and stores it in a database. The server generates a unique reservation ID and returns it to the user via the terminal.

[1773] Step 2:

[1774] On the day of the session, the user uses the microphone on the device to ask the expert a question. The device records the user's voice data and sends it to the server. The input at this time is the user's voice data.

[1775] Step 3:

[1776] The server converts the received voice data into text data using speech recognition software. This process uses the speech_recognition software. By converting the voice data into text, the content of the user's question is output as text data.

[1777] Step 4:

[1778] The server inputs the user's question, converted into text, into a generative AI model. GPT-2 is used as the generative AI model. The text data is passed as input to the AI ​​model, which generates an appropriate response text. The generative AI model generates a response using a prompt sentence based on the user's question. Specifically, a response based on the prompt sentence, "Please generate an appropriate reply to the question 'What caused this machine in the factory to stop?'" is output.

[1779] Step 5:

[1780] The generated response text is converted into voice data by the speech synthesis system in the server. For this purpose, we use the pyttsx3 software, which generates voice data from the text data and outputs the response voice.

[1781] Step 6:

[1782] The server transmits the generated voice data to the terminal, which receives the voice data and provides a response to the user through a speaker, where the output is a voice response that the user can hear.

[1783] Step 7:

[1784] After the session ends, the user sends a notification to the server via their device. The server receives this notification, analyzes the session, and creates a feedback report. This analysis and feedback includes usage data, user questions, and AI responses.

[1785] Step 8:

[1786] The server sends the created feedback report to the terminal, which displays the report to the user, allowing the user to use it for future learning and work improvement. The output at this time is the feedback report.

[1787] The above process realizes a system that allows users to quickly receive expert guidance and troubleshooting suggestions in real time using voice.

[1788] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1789] Based on the problem to be solved by the present invention, a specific embodiment for carrying out the invention in combination with an emotion engine will be described below.

[1790] 1. Reservation System Implementation

[1791] The user opens an interface on their terminal to reserve a session with an expert and enters the necessary reservation information, such as their ID, desired expert, and desired session date and time. The terminal sends this reservation information to the server's reservation API via a POST request. The server receives the reservation information and generates a unique reservation ID when saving it in the database. The server then notifies the user of the generated reservation ID.

[1792] 2. Session Format

[1793] At the start of a session, the user inputs a question or task by voice. The device records the user's voice and sends the voice data to the server. The server analyzes the received voice data, converts it into text using voice recognition technology, and then recognizes the user's emotions using an emotion engine. Based on the recognized emotions, the dialogue AI system generates an appropriate response. The generated text response is converted back into voice by the server and sent to the device. The device plays back this voice data and provides the response to the user.

[1794] 3. Feedback System Embodiments

[1795] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the user's learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can then review the feedback report and use it to plan their future learning.

[1796] Specific examples using emotion engines

[1797] Consider a scenario where, during a session, the user asks a question by voice, such as, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes the text data to the emotion engine. The emotion engine analyzes the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, the response generated might be, "Start with a basic tutorial, then hone your skills with practical projects." The server then converts this text back into audio data and provides it to the user.

[1798] Dynamic Adjustment Examples

[1799] The server evaluates the user's emotions in real time during the session and dynamically adjusts the expert's instruction. For example, if the user shows signs of anxiety, the expert will flexibly change the instruction to suit the situation. This allows the user to learn in a relaxed state and effectively acquire skills.

[1800] Example of a feedback report

[1801] After the session ends, the server receives a notification and analyzes the session content. This analysis includes the user's voice data, text data, and emotional history. Based on the analysis results, the server provides feedback such as, "The user has grasped basic knowledge, and in the future, more practical projects will further improve their skills." The feedback report is sent to the user, who can use it to plan their next learning.

[1802] In this way, a system can be implemented in which users can smoothly progress through learning by utilizing the emotion engine while receiving guidance from an expert, and can efficiently improve their skills.

[1803] The processing flow will be explained below.

[1804] Step 1:

[1805] The user opens an interface on the terminal for reserving a session with an expert and inputs necessary reservation information such as the user's ID, the desired expert, and the desired date and time of the session.

[1806] Step 2:

[1807] The terminal sends the entered reservation information to the server's reservation API via a POST request. The information sent includes the user ID, expert ID, desired date and time, etc.

[1808] Step 3:

[1809] The server checks the reservation information received through the booking API and saves it in the database, generating a unique reservation ID.

[1810] Step 4:

[1811] The server creates a response including the generated reservation ID and sends it to the user's terminal.

[1812] Step 5:

[1813] The terminal displays the reservation ID returned from the server and notifies the user that the reservation has been confirmed.

[1814] Step 6:

[1815] The user prepares to start the session at the fixed date and time. At the start of the session, the user inputs questions or tasks by voice.

[1816] Step 7:

[1817] The terminal records the user's voice and transmits the voice data to the server, which may be real-time or pre-recorded.

[1818] Step 8:

[1819] The server analyzes the received voice data and converts the voice data into text using speech recognition, which is then input to the emotion engine.

[1820] Step 9:

[1821] The server uses an emotion engine to recognize the user's emotions from their voice, and classifies the emotions into categories such as "happiness," "sadness," "anger," and "surprise."

[1822] Step 10:

[1823] The server inputs text data based on the user's emotions into the conversational AI system, which then generates an appropriate response. For example, if the user expresses "anxiety," the AI ​​will generate a more friendly and reassuring response.

[1824] Step 11:

[1825] The server converts the generated text response back into voice data, which is then sent to the user's terminal.

[1826] Step 12:

[1827] The device plays back the received audio data and provides the AI's response to the user, who can then continue with additional questions or challenges.

[1828] Step 13:

[1829] When a user wants to end a session, he or she sends a termination notice to the system. The termination notice is sent from the terminal to the server.

[1830] Step 14:

[1831] The server analyzes the session content based on the received end notification and evaluates the user's learning progress based on the audio data, text data, and emotion history collected during the session.

[1832] Step 15:

[1833] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's progress and advice on future learning.

[1834] Step 16:

[1835] The server then sends the created feedback report to the user, who then receives the report via email or in-app notification.

[1836] Step 17:

[1837] The user creates their next study plan based on the received feedback report. They check the report contents and decide on the next session reservation and study content.

[1838] Step 18:

[1839] The terminal then reopens the reservation interface and proceeds with the reservation of a new session. The system repeats the process, helping the user improve their skills.

[1840] Example 2

[1841] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1842] Conventional reservation systems have the drawback of requiring users to perform many manual operations when reserving a session with an expert, and making it difficult to manage reservation information. Furthermore, they are unable to grasp the user's emotions in real time during the session and dynamically adjust the expert's instruction content based on those emotions, resulting in insufficient learning effectiveness. Furthermore, feedback after the session is sometimes delayed, making it difficult to adequately support the user's learning progress.

[1843] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1844] In this invention, the server includes means for transmitting the user's reservation information to the server and saving the reservation information in a database, means for generating a specific reservation number and providing it to the user, means for transmitting voice data from the user to the server, means for converting the voice data to text, means for inputting the text data into an emotion analysis engine and recognizing emotions, means for generating a response using a generative AI model based on the recognized emotions, means for converting the generated response text into voice and providing it to the user, means for receiving a session end notification from the user and analyzing the session content and the user's emotion history to make a final evaluation, and means for creating a feedback report and sending it to the user. This enables the user to make a reservation smoothly, receive appropriate responses in real time during the session, and receive prompt feedback after the session ends.

[1845] A "user" is a person who utilizes the system to book and conduct sessions with experts.

[1846] An "expert" is someone who has extensive knowledge and experience in a particular field and who can provide guidance and advice to users.

[1847] A "server" is a computer system that receives information from users, processes the data, stores it in a database, and runs generative AI models.

[1848] A "terminal" is a device used by a user to access the system, and includes a PC, smartphone, tablet, etc.

[1849] A "generative AI model" is an artificial intelligence algorithm that generates optimal responses or predictions based on specific input data.

[1850] A "reservation number" is a unique identifier that allows a user to identify a reservation.

[1851] "Voice data" refers to data in which the user's voice is recorded in digital format.

[1852] "Text data" refers to character data obtained by analyzing and transcribing voice data.

[1853] An "emotion analysis engine" is an analysis device or program that analyzes a user's emotions (e.g., excitement, anxiety, joy, etc.) from input text data.

[1854] A "feedback report" is a report that includes an evaluation and suggestions for future learning, generated based on the user's performance and emotional history during the session.

[1855] This invention is a system that allows users to reserve sessions with experts, and receive feedback on the sessions. This system is composed of three entities: a server, a terminal, and a user, and operates as follows.

[1856] 1. Reservation System Implementation

[1857] To book a session with an expert, a user opens the device's interface and enters the necessary information, such as their ID, the preferred expert, and the desired date and time of the session. This interface is provided as a web page or mobile app. The device sends this reservation information to the server's reservation API via an HTTP POST request. The server executes an SQL query to store the received reservation information in a database and generates a unique reservation number. The server then returns a response including the generated reservation number to the device, and the user is notified. This allows the user to confirm that the reservation was successful.

[1858] 2. Session Format

[1859] At the start of a session, the user uses the voice input function to speak a question or task into the device. The device records the user's voice and sends the audio data to the server. This audio data is saved in an appropriate format (such as .wav or .mp3). The server analyzes the received audio data and converts it into text using speech recognition technology (e.g., Google Cloud Speech-to-Text). The server then passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. Based on the recognized emotion, a generative AI model (e.g., OpenAI GPT-3) generates a response. This response is again converted into speech by the server (e.g., Amazon Polly) and sent to the device. The device plays the audio data and provides the response to the user.

[1860] 3. Feedback System Embodiments

[1861] After the session ends, the user sends a completion notification to the system. The device then sends this completion notification to the server. The server analyzes the session content based on the received completion notification (e.g., using an NLP analysis tool) and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning.

[1862] Add specific examples to the description

[1863] Reservation system example

[1864] For example, if a user wants to reserve a session with an expert "Expert A" with ID "user123" for 15:00 on November 1, 2023, he or she enters this information into the terminal interface. The terminal sends this information to the server, which stores it in its database, generates a reservation number "reserve456" and notifies the user.

[1865] Session Example

[1866] For example, if a user asks, "What should I do to learn a new programming language?", the device records the question and sends it to a server. The server converts the speech to text using Google Cloud Speech-to-Text and analyzes sentiment using IBM Watson Tone Analyzer. OpenAI GPT-3 generates a response such as "Start with basic tutorials and hone your skills with hands-on projects," and Amazon Polly converts this text to speech and serves it to the user.

[1867] Examples of feedback systems

[1868] After the session ends, the server receives a notification from the user and analyzes the user's emotional history and session data. It then generates feedback to the user, such as "The user has grasped basic knowledge and would benefit from more practical projects."

[1869] Prompt Sentence Examples

[1870] "What should I do when learning a new programming language?"

[1871] Through each of these steps, users can enjoy a smooth booking, session, and feedback experience, and a system can be implemented that allows them to efficiently improve their skills while receiving expert guidance.

[1872] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1873] Step 1:

[1874] The user opens the device interface and enters information such as their ID, the preferred specialist, and the desired date and time of the session. This information is entered into the device as reservation information. The device temporarily stores this information and displays a confirmation screen to the user.

[1875] Step 2:

[1876] When the user confirms the reservation information and presses the send button, the device sends this reservation information to the server's reservation API via an HTTP POST request. The input data is the user ID, expert ID, and reservation date and time, and is sent to the server in JSON format.

[1877] Step 3:

[1878] The server analyzes the reservation information received from the terminal and generates an SQL query to store it in the database. Here, the user ID, expert ID, and reservation date and time are input data, and are stored in fields in the "Reservation" table. The server generates a unique reservation number and returns it as a response.

[1879] Step 4:

[1880] The terminal receives the response from the server and displays a confirmation message to the user, including the reservation number. This confirms that the reservation was successfully completed. The output is the reservation number and the confirmation message.

[1881] Step 5:

[1882] At the scheduled time, the user starts the session using the device. Using the voice input function, the user voice-enters questions and tasks into the device. This voice data becomes the input data.

[1883] Step 6:

[1884] The device records the user's voice and sends it to the server. The voice data is encoded in the appropriate format (e.g., .wav or .mp3). The data sent to the server is the binary data of the voice file.

[1885] Step 7:

[1886] The server analyzes the received voice data and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text). The input data is an audio file, and the output is text data. The server temporarily stores the text data.

[1887] Step 8:

[1888] The server passes the text data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The input is text data, and the output is the emotion recognition result (e.g., excitement, anxiety, etc.). The server temporarily stores the emotion recognition result.

[1889] Step 9:

[1890] The server inputs the recognized emotion and text data into a generative AI model (e.g., OpenAI GPT-3) to generate an appropriate response. The input data is text and emotion recognition results, and the output is a response text. The server temporarily stores the generated response text.

[1891] Step 10:

[1892] The server converts the generated response text into speech using speech synthesis technology (e.g., Amazon Polly). The input data is the response text, and the output is an audio file. The server then sends the audio data to the device.

[1893] Step 11:

[1894] The terminal plays back the voice data received from the server and provides a response to the user. The output is a voice response, allowing the user to experience a dialogue in real time.

[1895] Step 12:

[1896] After the session ends, the user sends a termination notification to the system. By clicking the end button, the terminal sends a termination notification to the server. The data sent is a termination notification flag.

[1897] Step 13:

[1898] The server receives the termination notification and analyzes the audio data, text data, and emotion history during the session (e.g., using an NLP analysis tool). The input data are audio data, text data, and emotion recognition results, and the output is the analysis results.

[1899] Step 14:

[1900] The server generates a feedback report based on the analysis results. The input data is the analysis results, and the output is the feedback report. The report contains information that is useful for the user's learning progress and future learning plans.

[1901] Step 15:

[1902] The server sends a feedback report to the user, and the sent data is the feedback report, which the user can check and use for future learning.

[1903] (Application example 2)

[1904] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1905] Conventional online session systems lack support for users to communicate effectively with experts. Furthermore, because they respond uniformly without considering the user's emotions, it is difficult to provide appropriate support for emotions such as anxiety and excitement. Furthermore, because feedback does not take into account the user's emotions or learning progress, it is difficult to create an effective learning plan.

[1906] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing user reservation information in a database, means for analyzing the user's voice input in real time based on the generated reservation information and recognizing emotions, means for generating an appropriate response to the voice input based on the emotion recognition, and means for dynamically adjusting and reproducing the response taking the user's emotions into consideration. This makes it possible to grasp the user's emotions in real time and provide an optimal response. In addition, by evaluating the user's learning progress and providing appropriate feedback, it is possible to create an effective learning plan.

[1907] "User" refers to a person who uses the system to hold a session with an expert.

[1908] An "expert" refers to a person or system that has specialized knowledge and skills in a particular field and that holds a session with a user to provide assistance.

[1909] A "session" refers to a period of interaction or instruction between a user and an expert for a specific purpose.

[1910] "Reservation information" refers to information provided by a user when reserving a session with an expert, and includes, for example, an ID, a desired expert, and a desired date and time for the session.

[1911] "Server" refers to a central processing unit that processes data and communicates between users and experts.

[1912] "Database" refers to a system that centrally manages and stores data such as reservation information.

[1913] A "reservation ID" is a unique identifier generated by the server and used to manage session reservations.

[1914] "Analyzing in real time" refers to a process of instantly analyzing voice data while the user is inputting voice data.

[1915] "Emotion recognition" refers to analyzing and identifying emotional states such as excitement, anxiety, and joy from a user's voice and text data.

[1916] "Dynamic response adjustment" refers to changing and optimizing expert guidance and system responses in real time according to the user's emotional state.

[1917] A "feedback report" is a report generated by the server after a session ends. It evaluates the user's learning progress and emotional history, and provides information that can be used as a reference for future learning plans.

[1918] The present invention provides a product purchasing support system in a virtual store that allows users to reserve a session with an expert and dynamically responds to user emotions. Specific embodiments of this system are described below.

[1919] 1. Reservation System Implementation

[1920] A user uses their device (e.g., smartphone, tablet, or computer) to reserve a session with an expert. The user opens the reservation interface and enters the necessary information, such as their ID, the desired expert, and the desired session date and time. The device then sends this reservation information to the server's reservation API via a POST request. The server stores the received reservation information in a database and generates a unique reservation ID. This reservation ID is notified to the user.

[1921] 2. Session Format

[1922] At the start of a session, the user vocally inputs a question or task related to the session content. The device records the user's voice and sends the voice data to the server. The server then uses speech recognition technology (e.g., Google Speech Recognition API) to convert the voice data into text. It then uses an emotion recognition engine (e.g., EmotionRecognizer library) to analyze and recognize the user's emotions from the text data. Based on the recognized emotions, the server's conversational AI system (e.g., generative AI model) generates an appropriate response. The generated text response is converted into voice data by the server (e.g., TextToSpeech library), sent to the device, and provided to the user.

[1923] 3. Feedback System Embodiments

[1924] After the session ends, the user sends a completion notification to the system. The device then sends this notification to the server. The server analyzes the session content based on the received notification and evaluates the learning progress based on the user's emotional history. A feedback report is created based on the analysis results, and the server sends this feedback report to the user. The user can review the feedback report and use it to plan their future learning. In this way, the user can effectively progress with their learning by utilizing the emotion engine while receiving guidance from an expert.

[1925] Specific examples

[1926] For example, consider a case where a user asks by voice, "What should I do when learning a new programming language?" The device records this question and sends it to the server. The server uses speech recognition technology to convert the question into text and passes this text data to an emotion recognition engine. The emotion recognition engine analyzes emotions from the user's voice and recognizes emotions such as "excitement" or "anxiety." Based on this, the conversational AI system generates an appropriate response to the user. For example, it might generate a response as text, such as "Start with a basic tutorial and then hone your skills with practical projects," and the server then converts this text back into audio data and provides it to the user.

[1927] Prompt Sentence Examples

[1928] User Question: "What should I do when learning a new programming language?"

[1929] Recognized emotion: "excitement"

[1930] This allows users to receive expert guidance in a way that suits their emotions, resulting in a more effective and personalized purchasing experience and learning.

[1931] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1932] Step 1:

[1933] The user opens the reservation interface and enters reservation information such as ID, desired expert, desired session date and time, etc. The terminal sends this input information to the server. The server stores the received reservation information in a database and generates a unique reservation ID. The server notifies the user of this reservation ID.

[1934] Step 2:

[1935] At the start of a session, the user enters a question or task by voice through the device. The device records the user's voice and sends the voice data to the server. The server receives the voice data and converts it into text using voice recognition technology.

[1936] Step 3:

[1937] The server inputs the converted text data into an emotion recognition engine, which analyzes the text data and identifies the user's emotion. The emotion recognition engine outputs the user's emotional state (e.g., excitement, anxiety).

[1938] Step 4:

[1939] The server generates an appropriate response using a generative AI model based on the recognized emotion. The generative AI model creates an optimal response text based on the user's question content and emotional state. The generative AI model outputs this response text.

[1940] Step 5:

[1941] The server uses the TextToSpeech library to convert the response text into audio data, which is then sent to the device, which plays the audio response, and the user receives the response as audio.

[1942] Step 6:

[1943] After the session ends, the user sends a notification to the server. The server receives the notification and analyzes the session contents, including the user's voice data, text data, and emotional history.

[1944] Step 7:

[1945] The server creates a feedback report based on the analysis results, which includes an evaluation of the user's learning progress and emotional state. The server then sends this feedback report to the terminal and provides it to the user.

[1946] The above are the specific processing steps of the system for realizing the present invention.

[1947] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1948] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1949] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1950] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1951] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1952] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1953] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1954] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1955] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1956] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1957] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1958] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1959] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1960] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1961] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1962] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1963] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1964] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1965] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1966] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1967] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1968] The following is further disclosed regarding the above embodiment.

[1969] (Claim 1)

[1970] a means for a user to book a session with an expert;

[1971] a means for transmitting the user's reservation information to a server, and the server storing the reservation information in a database;

[1972] a means for providing the generated reservation ID to the user;

[1973] A system including:

[1974] (Claim 2)

[1975] means for transmitting user voice data to a server, and the server converting the voice data into text;

[1976] A means for inputting text data into a generating artificial intelligence, and the generating artificial intelligence generating a response;

[1977] means for converting the generated response text into speech and providing it to the user;

[1978] 10. The system of claim 1, comprising:

[1979] (Claim 3)

[1980] a means for receiving a session end notification from a user, analyzing the session content, and creating a feedback report;

[1981] means for sending a feedback report to the user;

[1982] 10. The system of claim 1, comprising:

[1983] "Example 1"

[1984] (Claim 1)

[1985] a means for a user to schedule a meeting with an expert;

[1986] means for transmitting user reservation information to a communication device, and for the communication device to store the reservation information in a storage device;

[1987] means for providing the generated reservation identifier to a user;

[1988] the communication device includes means for converting voice data into text;

[1989] a means for inputting the generated text data into an artificial intelligence system, and the artificial intelligence system generating a response;

[1990] means for converting the generated response text into speech and providing it to the user;

[1991] A system including:

[1992] (Claim 2)

[1993] a means for receiving a meeting completion notification from a user, analyzing the meeting content, and creating an evaluation report;

[1994] means for transmitting the evaluation report to the user;

[1995] 10. The system of claim 1, comprising:

[1996] (Claim 3)

[1997] A means for analyzing the content of the user's meetings and evaluating the user's learning progress;

[1998] a means for generating an evaluation report including the generated evaluation results and transmitting the evaluation report to the user by the communication device;

[1999] 10. The system of claim 1, comprising:

[2000] "Application Example 1"

[2001] (Claim 1)

[2002] a means for a user to book a session with an expert;

[2003] a means for transmitting the user's reservation information to a server, and the server storing the reservation information in a database;

[2004] a means for providing the generated reservation ID to the user;

[2005] means for recognizing user voice data and converting it into text data;

[2006] A means for inputting the generated text data into a generating artificial intelligence, and the generating artificial intelligence generating a response;

[2007] means for providing the generated audio response to a user;

[2008] A system including:

[2009] (Claim 2)

[2010] A means for inputting text data into a generating artificial intelligence, and the generating artificial intelligence generating a response;

[2011] means for converting the generated response text into speech and providing it to the user;

[2012] 10. The system of claim 1, comprising:

[2013] (Claim 3)

[2014] a means for receiving a session end notification from a user, analyzing the session content, and creating a feedback report;

[2015] means for sending a feedback report to the user;

[2016] 10. The system of claim 1, comprising:

[2017] "Example 2: Combining Emotion Engines"

[2018] (Claim 1)

[2019] a means for a user to book a session with an expert;

[2020] a means for transmitting the user's reservation information to a server, and the server storing the reservation information in a database;

[2021] means for generating and providing a specific reservation number to a user;

[2022] A system including:

[2023] (Claim 2)

[2024] means for transmitting voice data from a user to a server;

[2025] a means for the server to convert the voice data into text;

[2026] A means for inputting text data into an emotion analysis engine to recognize emotions;

[2027] a means for generating a response using a generative AI model based on the recognized emotion;

[2028] means for converting the generated response text into speech and providing it to the user;

[2029] 10. The system of claim 1, comprising:

[2030] (Claim 3)

[2031] a means for receiving a session end notification from the user, analyzing the session content and the user's emotion history, and making a final evaluation;

[2032] means for generating and sending a feedback report to the user;

[2033] 10. The system of claim 1, comprising:

[2034] "Application example 2 when combining emotion engines"

[2035] (Claim 1)

[2036] a means for a user to book a session with an expert;

[2037] a means for transmitting the user's reservation information to a server, and the server storing the reservation information in a database;

[2038] a means for providing the generated reservation ID to the user;

[2039] means for analyzing a user's voice input in real time and recognizing emotions based on the generated reservation information;

[2040] means for generating an appropriate response to a speech input based on emotion recognition;

[2041] means for dynamically adjusting and replaying responses taking into account the user's emotions;

[2042] A system including:

[2043] (Claim 2)

[2044] means for transmitting user voice data to a server, and the server converting the voice data into text;

[2045] A means for inputting text data into a generating artificial intelligence, and the generating artificial intelligence generating a response;

[2046] means for converting the generated response text into speech and providing it to the user;

[2047] A means for analyzing the user's emotions in real time and dynamically adjusting the expert's instruction content based on the emotion history;

[2048] 10. The system of claim 1, comprising:

[2049] (Claim 3)

[2050] a means for receiving a session end notification from a user, analyzing the session content, and creating a feedback report;

[2051] means for sending a feedback report to the user;

[2052] A means for evaluating the user's learning progress based on the analysis results and providing feedback that will be useful for future learning planning;

[2053] 10. The system of claim 1, comprising: [Explanation of symbols]

[2054] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to book a session with an expert; a means for transmitting the user's reservation information to a server, and the server storing the reservation information in a database; a means for providing the generated reservation ID to the user; A system including:

2. means for transmitting user voice data to a server, and the server converting the voice data into text; A means for inputting text data into a generating artificial intelligence, and the generating artificial intelligence generating a response; means for converting the generated response text into speech and providing it to the user; The system of claim 1 , comprising:

3. a means for receiving a session end notification from a user, analyzing the session content, and creating a feedback report; means for sending a feedback report to the user; The system of claim 1 , comprising:

Images