system

Abstract

We provide the system. [Solution] User authentication means, A voice guidance system using generative artificial intelligence, A means of determining the processing flow based on user requirements, Means for obtaining and displaying contract information, A means of receiving user input information and updating the database, A means of notifying the user of the processing results, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional unmanned stores, there is no system that can appropriately obtain and process the information necessary when a user makes a change to the contract content or a new contract, which is inconvenient for users. Also, when there is a shortage of human resources, it takes time to respond, and there is a possibility that customer satisfaction will decrease. In order to solve such problems, the realization of a system that can smoothly perform contracts and information changes even without human intervention has been demanded.

Means for Solving the Problems

[0005] The present invention provides a system that includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, and a means for notifying the user of the processing results. Specifically, the user authenticates using a QR code (registered trademark), and then inputs their requirements via voice guidance using generative artificial intelligence. The processing flow is determined based on the requirements, and contract information is acquired and displayed. The user inputs the necessary information, the database is updated, and the results are notified. This realizes a system that enables quick and accurate contract procedures even in unmanned stores.

[0006] "User authentication means" refers to a means for a user to prove their legitimacy to the system, and includes a function that obtains user information using a QR code, sends it to the server, and performs authentication.

[0007] A "voice guidance system using generative artificial intelligence" is a means of receiving user requirements via voice, analyzing those requirements, and determining an appropriate processing flow, thereby enabling natural language dialogue with the user using generative artificial intelligence.

[0008] "Means for determining processing flows based on user requirements" refers to a method for analyzing user-inputted requirements and determining an appropriate processing flow corresponding to those requirements, automatically distributing flows using AI and server resources.

[0009] "Means for acquiring and displaying contract information" refers to means for acquiring user contract information from a database and displaying that information on the screen, and includes a function to display user-entered changes in an easy-to-understand manner.

[0010] "Means for receiving user input and updating the database" refers to means for receiving information entered by the user via screen or voice and updating the database information based on that information, and for performing accurate data updates through cooperation with the server.

[0011] "Means for notifying the user of the processing results" refers to means for notifying the user of the results of a procedure performed by the user via voice or screen, and includes a function to quickly inform the user whether the procedure was successful or not. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention provides a system that allows users to change contract information, sign up for new contracts, and consult about plans through self-service at unmanned stores. Specific embodiments of this system are described below.

[0034] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating the database, and means for notifying the user of the processing results.

[0035] First, the user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to a server to authenticate the user. If authentication is successful, the terminal activates the voice guidance system.

[0036] The voice guidance system uses generative artificial intelligence to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to a server, where the AI ​​analyzes the request and determines the appropriate processing flow.

[0037] Next, based on the user's requirements, the necessary contract information is retrieved from the server and displayed on the terminal screen. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the required information according to the displayed instructions, such as their new address, desired plan, and payment method.

[0038] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[0039] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which determines the appropriate processing flow. The user selects their desired plan and enters the necessary information. The server then updates the database and returns the processing result to the terminal. Finally, the terminal notifies the user that the new contract has been completed.

[0040] Thus, the present invention provides a system that enables users to perform self-services smoothly even in unmanned stores. This improves user convenience and reduces waiting times.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[0044] Step 2:

[0045] The device sends user information obtained from the QR code to the server.

[0046] Step 3:

[0047] The server checks the received user information against the database and performs authentication.

[0048] Step 4:

[0049] The server sends the authentication result back to the terminal.

[0050] Step 5:

[0051] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[0052] Step 6:

[0053] Through the voice guidance system, the terminal will ask the user, "How can I help you?"

[0054] Step 7:

[0055] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0056] Step 8:

[0057] The terminal sends the entered requirements to the server.

[0058] Step 9:

[0059] The server uses AI to analyze requirements and determine the appropriate processing flow.

[0060] Step 10:

[0061] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0062] Step 11:

[0063] The terminal displays the acquired contract information on its screen, and a voice guidance system explains the contents to the user.

[0064] Step 12:

[0065] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[0066] Step 13:

[0067] The terminal sends the information entered by the user to the server.

[0068] Step 14:

[0069] The server updates the database based on the received information.

[0070] Step 15:

[0071] Once the database update is complete, the server sends the processing results back to the terminal.

[0072] Step 16:

[0073] The terminal displays the processing result on the screen and notifies the user via the voice guidance system (e.g., "New contract completed").

[0074] Step 17:

[0075] The user presses the "Complete" button as a final confirmation to finish the process.

[0076] Step 18:

[0077] The device will end the session and display "Thank you for using our service" on the screen.

[0078] This series of steps will create a system that allows for quick and accurate contract procedures at unmanned stores.

[0079] (Example 1)

[0080] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0081] In modern society, users have a growing need to access services quickly and efficiently. However, traditional self-service stores often fail to provide adequate guidance and contract processing tailored to user needs, resulting in a poor user experience. Furthermore, understanding and completing complex procedures requires significant time and effort on the user's part, which is another major challenge.

[0082] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0083] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, a means for the voice guidance means to ask the user questions using voice messages generated by generative artificial intelligence, and a means for sending user input information to the server, where the server analyzes the input information using an AI model. This allows users to use the service easily and quickly, improving the user experience. Furthermore, complex procedures can be easily understood and completed, resulting in a reduction in time and effort.

[0084] "User authentication means" refers to the means used to verify a user's identity when they access a system.

[0085] "Generative artificial intelligence" refers to technologies that use artificial intelligence to generate text and speech in natural language processing and generative tasks.

[0086] "Voice guidance means" refers to a means of providing guidance or instructions to users using voice messages.

[0087] "Means for determining the processing flow" refers to the means by which the system selects and executes appropriate processing procedures based on user requirements.

[0088] "Means for obtaining and displaying contract information" refers to means for obtaining necessary contract information from a database and displaying it so that users can verify it.

[0089] "A means of receiving user input and updating the database" refers to a means by which the system receives information entered by the user and reflects it in the database.

[0090] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing has been completed.

[0091] A "voice message" refers to voice guidance or instructions generated by generative artificial intelligence and provided to the user through speech synthesis technology.

[0092] A "server" is a computer system that processes user input information and performs tasks such as providing necessary data and updating databases.

[0093] An "AI model" is an artificial intelligence algorithm or network trained to perform data analysis and generation tasks.

[0094] This invention provides a system that allows users in unmanned stores to change contract information, enter into new contracts, and consult about plans through self-service. A specific embodiment of this system is described below.

[0095] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, and means for notifying the user of processing results.

[0096] First, upon arriving at the unmanned store, the user scans their QR code using the scanner on the provided terminal. The terminal then sends the information obtained from the QR code to a server to authenticate the user. Specifically, the terminal analyzes the data from the QR code and sends it to the server via wireless or wired communication. The server searches its database to verify the authentication information. For example, MySQL® can be used as the database.

[0097] Upon successful authentication, the terminal activates the voice guidance system. The voice guidance system uses generative artificial intelligence (e.g., OpenAI®'s GPT-4®) to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the request and determines the appropriate processing flow.

[0098] Based on the user's requirements, the server retrieves the necessary contract information and displays it on the terminal screen. For example, it might use MongoDB to retrieve contract plan information. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the necessary information according to the displayed instructions, such as their new address, desired plan, and payment method.

[0099] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[0100] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system, asking, "How can I help you?" The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which uses the generative AI model GPT-4 to determine the appropriate processing flow. The user selects their desired plan and enters their address, payment method, etc. The server updates the database and returns the processing result to the terminal. The terminal notifies the user, "Your contract is complete. Thank you for using our service."

[0101] Examples of prompt statements to input into a generative AI model are as follows:

[0102] "The user wishes to sign up for a new internet connection. Please enter the required information."

[0103] This system allows users to smoothly perform self-services in unmanned stores. This improves user convenience and enables the completion of procedures quickly and efficiently.

[0104] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0105] Step 1: User Authentication

[0106] The user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to the server to authenticate the user.

[0107] Input: The QR code held up to the scanner on the user's device.

[0108] Operation: The device scans the QR code, analyzes the data, and extracts information such as the user ID.

[0109] Data processing: Convert QR code data into text information (user ID, membership number, etc.).

[0110] Output: Text information is sent to the server.

[0111] Specific operation: The terminal sends scan data to the server via wireless or wired connection.

[0112] The server searches the database based on the received information and verifies the authentication credentials.

[0113] Input: User information sent from the device.

[0114] Data processing: Issue database queries to retrieve user information.

[0115] Output: Authentication result (success or failure).

[0116] Step 2: Activate the voice guidance system

[0117] Upon successful authentication, the terminal will activate the voice guidance system.

[0118] Operation: The device calls a speech synthesis API (e.g., Google® Text-to-Speech) to generate a voice message.

[0119] Input: Authentication result status (success).

[0120] Output: A voice message "How can I help you?" is generated and played for the user.

[0121] Specific operation: The device uses a speech synthesis API to generate a voice message and plays it through the speaker.

[0122] Step 3: Enter requirements

[0123] Users enter their requirements via voice or touch panel.

[0124] Input: Requirements entered by the user via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0125] Operation: The terminal receives user input and converts it into text data.

[0126] Output: Requirements information as text data.

[0127] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice input into text data.

[0128] Step 4: Analyze requirements and determine the processing flow.

[0129] The terminal sends requirements information to the server, and the server-side AI analyzes the requirements and determines the appropriate processing flow.

[0130] Input: Text data of requirements information.

[0131] Operation: The server receives the requirements information and inputs it as prompts to a generative AI model (e.g., GPT-4).

[0132] Data processing: The AI ​​model generates a processing flow based on prompts.

[0133] Output: Processing flow of the analysis results.

[0134] Specific operation: The server calls a generative AI model and retrieves the analysis results.

[0135] Step 5: Obtain and present contract information

[0136] The server retrieves the necessary contract information based on the user's requirements and displays it on the terminal screen.

[0137] Input: Database query based on the processing flow.

[0138] Operation: The server queries the database and retrieves the necessary contract information (e.g., plan, fees, terms and conditions, etc.).

[0139] Data processing: Database queries are executed to extract contract information.

[0140] Output: Display data of contract information.

[0141] Specific operation: Data obtained from the server is sent to the terminal, and the terminal displays that information on its screen.

[0142] Step 6: Entering Information

[0143] The user follows the on-screen instructions and enters the required information (address, plan selection, payment method, etc.).

[0144] Input: Information entered by the user.

[0145] Operation: The terminal receives user input information and sends it to the server.

[0146] Output: Input information is passed to the server.

[0147] Specific operation: The device provides information as an input form and accepts input via touch panel or keyboard.

[0148] Step 7: Data Update

[0149] The server updates the database using the user's input information.

[0150] Input: User-provided information.

[0151] Operation: The server executes an update query against the database.

[0152] Data calculation: Update the corresponding record in the database.

[0153] Output: Database update results.

[0154] Specific operation: The server issues SQL queries to update database information (e.g., MongoDB or MySQL).

[0155] Step 8: Notification of processing results

[0156] Once the update is complete, the server returns the processing results to the terminal.

[0157] Input: Database update results.

[0158] Operation: The server retrieves the update results and sends them back to the terminal.

[0159] Output: Processing result (success, failure, etc.).

[0160] Specific actions: The terminal displays the result on the screen, and the voice guidance system notifies the user, "Your contract has been completed. Thank you for using our service."

[0161] By following the steps above in sequence, users can smoothly utilize self-services in unmanned stores.

[0162] (Application Example 1)

[0163] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0164] In modern society, users are required to quickly and efficiently change contract information, sign up for new contracts, and consult about plans. However, conventional systems often involve cumbersome procedures and long waiting times. In particular, in unmanned stores, it is difficult for users to perform these operations easily through self-service, and there is a need for effective means to improve the user experience. Furthermore, there is a need for a system that can be easily operated using smartphones and other mobile devices.

[0165] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0166] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, an application means installed on a smartphone, a means for collecting user requirements using voice recognition technology, and a means for determining a processing flow based on the requirements and displaying information. As a result, users can smoothly perform self-service even in unmanned stores using their smartphones, thereby reducing waiting times and improving convenience.

[0167] "User authentication means" refers to the means used to authenticate a user when they access a system.

[0168] "Generative artificial intelligence" refers to artificial intelligence that has the ability to automatically generate text and speech.

[0169] "Voice guidance means" refers to a means of guiding users using voice.

[0170] A "processing flow based on user requirements" is a means of determining the appropriate processing flow according to the requirements entered by the user.

[0171] "Contract information" refers to information about the services and plans that a user has contracted for.

[0172] "User input information" refers to all information that a user enters into the system.

[0173] "Methods for updating the database" refer to methods for updating the contents of a database based on information entered by the user.

[0174] "Means of notifying the user of the processing results" refers to means of informing the user that the processing has been completed.

[0175] "Applications installed on smartphones" refers to applications that are installed on and used on smartphones.

[0176] "Speech recognition technology" is a technology that converts speech into text.

[0177] "Means for displaying information" refers to means of displaying information on a screen or other device.

[0178] The present invention is a system for users to perform self-services using a smartphone in an unmanned store, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, means for an application installed on the smartphone, means for collecting user requirements using voice recognition technology, and means for determining a processing flow based on the requirements and displaying information.

[0179] A specific embodiment of this system will be described.

[0180] First, the user arrives at the unmanned store and scans a provided QR code using their smartphone. The user authentication system is activated, sending the information obtained from the QR code to a server to authenticate the user. If this authentication is successful, the smartphone application activates a voice guidance system using generative artificial intelligence and asks, "How can I help you?" This prompt is generated using a generative AI model.

[0181] The user inputs their requirements via voice, such as "I want to sign up for a new internet connection." Speech recognition technology, such as the Google Speech-to-Text API, converts the voice into text, and this information is sent to the server. The server analyzes the received text information and determines the processing flow based on the user's requirements.

[0182] The server retrieves contract information according to a predetermined processing flow and displays it on the smartphone screen. At this time, a voice guidance system using generative artificial intelligence guides the user through the displayed information in an easy-to-understand manner. For example, a prompt such as "Please select from these plans" may be displayed.

[0183] The user reviews the contract information and plan details displayed on the screen and enters the necessary information. The entered information is sent back to the server, which uses it to update the database. Once the update is complete, the server returns the processing result to the smartphone and notifies the user via voice guidance. For example, a prompt message such as "Your contract is complete. Thank you for using our service" may be displayed.

[0184] Here, smartphones are primarily used as hardware, and the software utilizes generative artificial intelligence models (e.g., OpenAI's GPT-4) and speech recognition technology (e.g., Google Speech-to-Text API). On the server side, cloud databases (e.g., AWS® RDS or Google Cloud SQL) are used to manage contract and user information.

[0185] As a concrete example, let's look at the procedure for signing up for a new line at an unmanned store. First, the user scans a QR code, and a smartphone application launches. A voice guidance system asks, "How can I help you?", and the user replies, "I would like to sign up for a new line." The server analyzes the requirements and displays an appropriate plan, and the user enters the necessary information. After that, the contract information is updated in the database, and the result is notified to the user. This allows the user to perform the service smoothly on their own.

[0186] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0187] Step 1:

[0188] The user arrives at the unmanned store and scans a provided QR code using their smartphone. The smartphone obtains a user ID from the scanned QR code and sends it to the server as authentication information. The server uses the received user ID to look up information in its database and authenticates the user. If authentication is successful, the server sends an authentication success signal to the smartphone.

[0189] Step 2:

[0190] When the smartphone receives a successful authentication signal, a voice guidance system using generative artificial intelligence is activated. The generative AI generates the prompt "How can I help you?" and asks the user through the smartphone's voice output device. The user responds by voice, "I would like to sign up for a new line."

[0191] Step 3:

[0192] Smartphone speech recognition technology (e.g., Google Speech-to-Text API) converts the user's speech into text. This text data is sent to the server as the user's requirements. The server analyzes the received text data and determines the appropriate processing flow.

[0193] Step 4:

[0194] The server retrieves the necessary contract information from the database based on the user's requirements. The retrieved contract information is sent to the smartphone, and the contract information and plan details are displayed on the smartphone screen. Generative artificial intelligence generates prompt messages such as "Please select from these plans" and provides voice guidance.

[0195] Step 5:

[0196] The user checks their smartphone screen and enters their desired plan and necessary information. This input data is sent from the smartphone to the server. The server updates the database based on the received data and confirms that the update is complete.

[0197] Step 6:

[0198] The server records the completion of the database update and sends the result to the smartphone. The smartphone displays the processing result on the screen, and a generative artificial intelligence generates a prompt message such as "Your contract has been completed. Thank you for using our service," which is then notified to the user via voice. This allows the user to confirm the completion of the new contract.

[0199] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0200] This invention combines an emotion engine with a system that allows users to perform self-service tasks such as changing contract information, signing up for new contracts, and consulting on plans in unmanned stores. Specific embodiments of this system are described below.

[0201] This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[0202] First, the user arrives at the unmanned store terminal and scans a QR code using the terminal's scanner. The terminal sends the user information obtained from the QR code to a server for user authentication. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence.

[0203] Next, the voice guidance system, using an emotion engine, asks the user, "How can I help you?" The emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state. Based on the analyzed emotional information, the voice guidance system adjusts its response to the user's input requirements.

[0204] The user enters their requirements via voice or touch panel. For example, the requirement might be, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow.

[0205] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and the voice guidance system explains its contents to the user. During this process, the emotion engine continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[0206] The user follows the on-screen instructions and enters the necessary information (e.g., new address, desired plan, payment method, etc.). The terminal sends the information entered by the user to the server, which then updates its database based on that information.

[0207] Once the update is complete, the server sends the processing results back to the terminal. The terminal displays the processing results on the screen and notifies the user through the voice guidance system. For example, it might notify the user that "New contract completed."

[0208] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The emotion engine analyzes the user's emotional state and adjusts its response accordingly. The user replies, "I would like to sign up for a new line contract," and enters the necessary information. The server updates the database and sends the processing results back to the terminal. Finally, the terminal notifies the user that the new contract is complete, and the emotion engine displays a message to provide reassurance.

[0209] By combining these emotion engines, a system can be realized that allows users to use their services more comfortably. Flexible responses tailored to the user's emotional state become possible, leading to an expected improvement in service quality.

[0210] The following describes the processing flow.

[0211] Step 1:

[0212] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[0213] Step 2:

[0214] The device sends user information obtained from the QR code to the server.

[0215] Step 3:

[0216] The server checks the received user information against the database and performs authentication.

[0217] Step 4:

[0218] The server sends the authentication result back to the terminal.

[0219] Step 5:

[0220] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[0221] Step 6:

[0222] Through the voice guidance system, the terminal asks the user, "How can I help you?" At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[0223] Step 7:

[0224] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0225] Step 8:

[0226] The terminal sends the entered requirements to the server.

[0227] Step 9:

[0228] The server uses AI to analyze requirements and determine the appropriate processing flow.

[0229] Step 10:

[0230] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0231] Step 11:

[0232] The terminal displays acquired contract information on its screen, and a voice guidance system explains its contents to the user. Simultaneously, an emotion engine continuously monitors the user's emotions and provides messages to encourage relaxation as needed.

[0233] Step 12:

[0234] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[0235] Step 13:

[0236] The terminal sends the information entered by the user to the server.

[0237] Step 14:

[0238] The server updates the database based on the received information.

[0239] Step 15:

[0240] Once the database update is complete, the server sends the processing results back to the terminal.

[0241] Step 16:

[0242] The terminal displays the processing result on the screen and notifies the user through the voice guidance system (e.g., "Your new contract has been completed"). The emotion engine also provides messages to enhance user satisfaction based on the processing result (e.g., "Thank you for using our service. Your new contract has been successfully completed.").

[0243] Step 17:

[0244] The user presses the "Complete" button as a final confirmation to finish the process.

[0245] Step 18:

[0246] The device ends the session and displays "Thank you for using our service" on the screen. Additionally, an emotion engine monitors the user's facial expressions to confirm their satisfaction.

[0247] This series of steps, combined with an emotion engine, allows users to smoothly complete contract procedures even in unmanned stores while simultaneously receiving emotional support. This significantly improves user convenience and satisfaction.

[0248] (Example 2)

[0249] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0250] In self-service environments at unmanned stores, it is essential to provide a smooth and efficient service that minimizes user stress while allowing for changes to contract information, new contracts, and plan consultations. Furthermore, improving service quality by appropriately responding to users' emotional states is a key challenge.

[0251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0252] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine that analyzes the user's emotions and adjusts responses according to the user's emotional state. As a result, users can use self-service in unmanned stores without feeling stressed and while receiving appropriate responses.

[0253] "User authentication means" refers to a method for a user to identify themselves at a terminal in an unmanned store and to confirm their access rights to the system.

[0254] "Voice guidance using generative artificial intelligence" refers to a method that utilizes generative artificial intelligence technology to provide guidance and answer questions through natural dialogue with the user.

[0255] "Means for determining a processing flow based on requirements" refers to the means by which the system determines the appropriate processing procedure according to the requirements entered by the user.

[0256] "Means for acquiring and displaying contract information" refers to means for retrieving necessary contract information from a database and presenting it visually to the user.

[0257] "Means for receiving user input information and updating the database" refers to means of receiving information entered by a user into a terminal and updating the database based on that information.

[0258] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing is completed.

[0259] An "emotion engine" is an engine that analyzes the user's voice tone, word choice, facial expressions, etc., to recognize the user's emotional state and adjust its response accordingly.

[0260] "Means of reading QR codes" refers to the means of reading a QR code presented by a user and obtaining its information.

[0261] "Method for sending to a server and performing authentication" refers to a method for sending the read information to a server and for the server to perform user authentication.

[0262] "Means for activating the voice guidance system" refers to the means for starting a voice guidance system using generative artificial intelligence after successful user authentication.

[0263] "Means for adjusting the voice guidance system's response based on the user's emotional state" refers to means for appropriately changing the guidance and responses of the voice guidance system according to the user's emotional state, as analyzed by the emotion engine.

[0264] Modes for carrying out the invention

[0265] This invention relates to a system for enabling users in unmanned stores to utilize self-services and smoothly change contract information, enter into new contracts, and consult about plans. This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[0266] The program for this system is processed as follows:

[0267] The hardware used includes terminals, servers, QR code readers, touch panels, microphones, speakers, and cameras installed in unmanned stores. The software includes generative artificial intelligence models, speech recognition and synthesis systems, database management systems, and sentiment analysis engines.

[0268] First, when a user arrives at the unmanned store, they scan a QR code on the terminal's scanner to authenticate themselves. The terminal sends the user information obtained from the QR code to a server, which then authenticates the user by referring to a database. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence and asks the user what they need. At this time, an emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[0269] The user enters their requirements via voice or touch panel. For example, they might enter a requirement such as, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow. The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and a voice guidance system explains the contents to the user. The emotion engine continuously monitors the user's emotional state and provides messages to promote relaxation and appropriate advice as needed.

[0270] The user follows the on-screen instructions and enters the necessary information. For example, they enter their new address, desired plan, and payment method. The terminal sends the information entered by the user to the server, which then updates its database based on that information. Once the update is complete, the server sends the processing result back to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system. For example, it might notify the user, "New contract completed," and the emotion engine might display a message to provide reassurance.

[0271] As a concrete example, a possible prompt message for a new line contract might be: "A user who wants to sign up for a new line contract has arrived at the unmanned store and scanned the QR code. What should they do next?"

[0272] This system allows users to comfortably utilize the service without experiencing stress. Furthermore, it enables flexible responses tailored to the user's emotional state, which is expected to improve the quality of the service.

[0273] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0274] Detailed explanation of the processing steps

[0275] Step 1: User Authentication Procedure

[0276] Step 1-1: The user scans the QR code.

[0277] The user arrives at the unmanned store and holds the QR code up to the terminal's scanner.

[0278] Input: QR code information

[0279] Output: QR code data

[0280] Step 1-2: The device scans the QR code.

[0281] The terminal reads the QR code and obtains its data.

[0282] Input: QR code data

[0283] Output: Obtained user information

[0284] Steps 1 - 3: The terminal sends the obtained user information to the server.

[0285] The terminal sends the obtained user information to the server.

[0286] Input: Obtained user information

[0287] Output: User information sent to the server

[0288] Steps 1 - 4: The server performs user authentication.

[0289] The server accesses the database based on the received user information and performs user authentication.

[0290] Input: User information sent to the server

[0291] Data processing / Calculation: User authentication by database search

[0292] Output: Authentication result (success / failure)

[0293] Step 2: Voice guidance and requirement input

[0294] Steps 2 - 1: The terminal activates the voice guidance system.

[0295] The terminal confirms successful authentication and activates the voice guidance system using a generative artificial intelligence.

[0296] Input: Authentication result (success)

[0297] Output: Activation of the voice guidance system

[0298] Step 2-2: The terminal asks the user for requirements

[0299] The terminal asks the user "What can I do for you?" through the voice guidance system. At this time, the emotion engine analyzes the user's voice tone, diction, and expression to recognize the emotional state.

[0300] Input: Voice recognition, voice tone, diction, expression

[0301] Data processing / operation: Emotion analysis by the emotion engine

[0302] Output: The user's emotional state and encouragement for requirement input

[0303] Step 3: Processing of requirements

[0304] Step 3-1: The user inputs requirements

[0305] The user inputs requirements such as "I want to sign up for a new line" via voice or touch panel.

[0306] Input: User requirements

[0307] Output: Input requirement information

[0308] Step 3-2: The terminal sends the requirement information to the server

[0309] The terminal sends the input requirement information to the server.

[0310] Input: Input requirement information

[0311] Output: Requirement information sent to the server

[0312] Step 3-3: The server analyzes the requirements

[0313] The server analyzes the information it receives and determines the appropriate processing flow.

[0314] Input: Requirements information sent to the server

[0315] Data processing / calculation: AI-based requirements analysis and processing flow determination for servers.

[0316] Output: Determined processing flow

[0317] Step 3-4: The server retrieves the contract information.

[0318] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0319] Input: Determined processing flow

[0320] Data processing / calculation: Retrieving contract information from databases

[0321] Output: Retrieved contract information

[0322] Step 4: Confirm and enter contract information

[0323] Step 4-1: The device displays the contract information.

[0324] The terminal displays acquired contract information on the screen, and a voice guidance system explains the contents to the user. An emotion engine continuously monitors the user's emotional state.

[0325] Input: Retrieved contract information

[0326] Output: Contract information displayed on the screen, voice guidance to the user.

[0327] Step 4-2: The user enters the required information.

[0328] The user follows the on-screen instructions and enters information such as their new address, desired plan, and payment method.

[0329] Input: Information required for the contract (address, plan, payment method)

[0330] Output: Input information

[0331] Step 5: Data updates and notifications

[0332] Step 5-1: The terminal sends the input information to the server.

[0333] The terminal sends the information entered by the user to the server.

[0334] Input: Information entered by the user

[0335] Output: Information sent to the server

[0336] Step 5-2: The server updates the database.

[0337] The database is updated based on the information received by the server.

[0338] Input: Information sent to the server

[0339] Data processing / calculations: Database updates

[0340] Output: Update complete result

[0341] Step 5-3: The server sends the update results to the terminal.

[0342] The server sends the result of the database update to the terminal.

[0343] Input: Update complete result

[0344] Output: Update results sent to the terminal

[0345] Step 5-4: The device notifies the user.

[0346] The terminal displays the processing result on the screen and notifies the user via the voice guidance system that "the new contract has been completed." The emotion engine then displays a message to provide reassurance.

[0347] Input: Update results sent to the device

[0348] Output: Processing results displayed on the screen, notifications to the user

[0349] By processing each step in this manner, users can utilize their own services efficiently and without stress.

[0350] (Application Example 2)

[0351] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0352] Modern self-service systems in unmanned stores and physical stores often provide only standardized responses without considering the user's emotional state, resulting in a limited user experience. This can lead to user stress or inadequate service. The present invention aims to solve this problem, enabling flexible responses based on the user's emotional state and improving the quality of service.

[0353] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0354] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, an emotion analysis means, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion feedback means. This enables the user to receive appropriate responses according to their emotional state.

[0355] "User authentication means" refers to the means used to verify that a user is legitimate when accessing a system.

[0356] "Voice guidance means using generative artificial intelligence" refers to a means of providing voice guidance using generative artificial intelligence.

[0357] An "emotion analysis method" is a means of recognizing a user's emotional state by analyzing their voice tone and facial expressions.

[0358] "Means for determining a processing flow based on user requirements" refers to a means of determining the optimal processing flow based on the requirements entered by the user.

[0359] "Means for acquiring and displaying contract information" refers to means that help acquire necessary contract information and display it to the user.

[0360] "A means of receiving user input information and updating the database" refers to a means of receiving information entered by a user and updating the database based on that information.

[0361] "Means for notifying the user of processing results" refers to the means of notifying the user of processing results obtained from the server.

[0362] An "emotional feedback method" is a means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[0363] "Means of reading QR codes" refers to any means of reading a QR code.

[0364] "A means of sending scanned information to a server and performing authentication" refers to a method of sending information obtained from a QR code to a server and performing user authentication based on that information.

[0365] "A means of receiving user requirements input by voice" refers to a means of receiving requirements entered by the user via voice.

[0366] "Means for analyzing requirements input and instructing the server on the appropriate processing flow" refers to means for analyzing the user's requirements input and instructing the server on the appropriate processing flow based on that analysis.

[0367] "Means for analyzing the user's voice tone and facial expressions" refers to means of analyzing the user's voice and facial expressions to recognize their emotional state.

[0368] "Means for displaying relaxation messages based on the user's emotional state" refers to means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[0369] The present invention is a system that allows users to change contract information, enter into new contracts, consult about plans, etc., through self-service at physical stores, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for emotion analysis, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, and means for emotion feedback.

[0370] When a user accesses a device, they first scan a QR code using the device's scanner. The scanned information is sent to a server for user authentication. This authentication process uses a standard QR code reader and an authentication system running on the server side.

[0371] Upon successful authentication, a voice guidance system using generative artificial intelligence will be activated. This system provides voice guidance as its user interface. The voice guidance uses the speech synthesis software "pyttsx3" and provides prompts to ask for the user's requirements.

[0372] Simultaneously, emotion analysis tools analyze the user's voice tone and facial expressions to recognize their emotional state. Emotion analysis utilizes facial expression recognition libraries such as "DeepFace" and voice analysis software.

[0373] When a user enters their requirements, such as "I want to sign up for a new internet connection," via voice or touch panel, that information is sent to a server. The server analyzes the requirements and determines the optimal processing flow. An AI model is used for this analysis.

[0374] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen and explains it to the user verbally. During this process, the emotion analysis system continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[0375] The user enters necessary information such as their new address, desired plan, and payment method. This information is sent to the server, and the database is updated. The server returns the processing result to the device, which displays the result on the screen and notifies the user via audio.

[0376] Furthermore, the emotional feedback system displays relaxing messages based on the user's emotional state, providing a sense of security. This allows users to complete procedures stress-free.

[0377] Specific example:

[0378] If a user says "I want to sign up for a new smartphone contract" at a physical store, the emotion analysis system can detect the user's excitement and begin the process with a message like, "It's great that you now have a new smartphone to choose from!"

[0379] Example of a prompt:

[0380] "Please suggest how to respond if a customer appears anxious."

[0381] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0382] Step 1:

[0383] The user scans a QR code with their device. The input is the data in the QR code, and the output is the user information contained in the QR code. Specifically, the device's camera captures the QR code, and the digital data is analyzed to extract the information.

[0384] Step 2:

[0385] The device sends the acquired user information to the server, which then performs authentication. The input is user information obtained from a QR code, and the output is the authentication result (success or failure). Specifically, the server verifies the user's legitimacy by comparing it with a database.

[0386] Step 3:

[0387] Once authentication on the server is successful, the terminal activates a voice guidance system using generative artificial intelligence. The input is a signal indicating successful authentication, and the output is the start of voice guidance. Specifically, the terminal uses the speech synthesis software "pyttsx3" to ask the user, "How can I help you?"

[0388] Step 4:

[0389] The voice guidance system receives user requirements via voice. The input is the user's voice, and the output is the recognized requirements. Specifically, voice recognition software converts the user's voice into text data.

[0390] Step 5:

[0391] The terminal sends the acquired voice data to the server, which analyzes the requirements and determines the processing flow. The input is text data containing the user's requirements, and the output is an appropriate processing flow. Specifically, the server uses an AI model to analyze the requirements and select the processing flow.

[0392] Step 6:

[0393] The server retrieves the necessary contract information from the database and sends it to the terminal. The input is a query containing the required contract information, and the output is the contract information itself. Specifically, the server extracts the contract information from the database and sends it back to the terminal.

[0394] Step 7:

[0395] The terminal displays contract information on its screen, and a voice guidance system explains the contents to the user. The input is contract information, and the output is voice guidance and screen display. Specifically, the terminal displays contract information on a GUI, and speech synthesis software provides explanations.

[0396] Step 8:

[0397] The emotion analysis system analyzes the user's voice tone and facial expressions to monitor their emotional state. Inputs include camera footage and microphone audio, and output is emotional state data. Specifically, emotion analysis is performed using the DeepFace library and voice analysis software.

[0398] Step 9:

[0399] The user enters the necessary information, and that data is sent from the device to the server. The input includes information provided by the user (address, plan, payment method, etc.), and the updated data is saved to the database as output. Specifically, the device accepts user input via text boxes or dropdown menus.

[0400] Step 10:

[0401] The server updates the database and sends the results back to the terminal. The input is the information to be updated, and the output is the updated result. Specifically, the server updates the database records and sends the results to the terminal.

[0402] Step 11:

[0403] The device notifies the user of the processing results, and if necessary, an emotional feedback mechanism displays a relaxation message. The input consists of processing results and emotional state data, while the output is a notification to the user and emotional feedback. Specifically, the device displays the processing results on the screen and plays a message using speech synthesis software.

[0404] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0405] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0406] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0407] [Second Embodiment]

[0408] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0409] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0410] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0411] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0412] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0414] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0415] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0416] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0417] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0418] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0419] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0420] This invention provides a system that allows users to change contract information, sign up for new contracts, and consult about plans through self-service at unmanned stores. Specific embodiments of this system are described below.

[0421] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating the database, and means for notifying the user of the processing results.

[0422] First, the user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to a server to authenticate the user. If authentication is successful, the terminal activates the voice guidance system.

[0423] The voice guidance system uses generative artificial intelligence to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to a server, where the AI ​​analyzes the request and determines the appropriate processing flow.

[0424] Next, based on the user's requirements, the necessary contract information is retrieved from the server and displayed on the terminal screen. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the required information according to the displayed instructions, such as their new address, desired plan, and payment method.

[0425] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[0426] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which determines the appropriate processing flow. The user selects their desired plan and enters the necessary information. The server then updates the database and returns the processing result to the terminal. Finally, the terminal notifies the user that the new contract has been completed.

[0427] Thus, the present invention provides a system that enables users to perform self-services smoothly even in unmanned stores. This improves user convenience and reduces waiting times.

[0428] The following describes the processing flow.

[0429] Step 1:

[0430] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[0431] Step 2:

[0432] The device sends user information obtained from the QR code to the server.

[0433] Step 3:

[0434] The server checks the received user information against the database and performs authentication.

[0435] Step 4:

[0436] The server sends the authentication result back to the terminal.

[0437] Step 5:

[0438] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[0439] Step 6:

[0440] Through the voice guidance system, the terminal will ask the user, "How can I help you?"

[0441] Step 7:

[0442] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0443] Step 8:

[0444] The terminal sends the entered requirements to the server.

[0445] Step 9:

[0446] The server uses AI to analyze requirements and determine the appropriate processing flow.

[0447] Step 10:

[0448] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0449] Step 11:

[0450] The terminal displays the acquired contract information on its screen, and a voice guidance system explains the contents to the user.

[0451] Step 12:

[0452] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[0453] Step 13:

[0454] The terminal sends the information entered by the user to the server.

[0455] Step 14:

[0456] The server updates the database based on the received information.

[0457] Step 15:

[0458] Once the database update is complete, the server sends the processing results back to the terminal.

[0459] Step 16:

[0460] The terminal displays the processing result on the screen and notifies the user via the voice guidance system (e.g., "New contract completed").

[0461] Step 17:

[0462] The user presses the "Complete" button as a final confirmation to finish the process.

[0463] Step 18:

[0464] The device will end the session and display "Thank you for using our service" on the screen.

[0465] This series of steps will create a system that allows for quick and accurate contract procedures at unmanned stores.

[0466] (Example 1)

[0467] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0468] In modern society, users have a growing need to access services quickly and efficiently. However, traditional self-service stores often fail to provide adequate guidance and contract processing tailored to user needs, resulting in a poor user experience. Furthermore, understanding and completing complex procedures requires significant time and effort on the user's part, which is another major challenge.

[0469] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0470] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, a means for the voice guidance means to ask the user questions using voice messages generated by generative artificial intelligence, and a means for sending user input information to the server, where the server analyzes the input information using an AI model. This allows users to use the service easily and quickly, improving the user experience. Furthermore, complex procedures can be easily understood and completed, resulting in a reduction in time and effort.

[0471] "User authentication means" refers to the means used to verify a user's identity when they access a system.

[0472] "Generative artificial intelligence" refers to technologies that use artificial intelligence to generate text and speech in natural language processing and generative tasks.

[0473] "Voice guidance means" refers to a means of providing guidance or instructions to users using voice messages.

[0474] "Means for determining the processing flow" refers to the means by which the system selects and executes appropriate processing procedures based on user requirements.

[0475] "Means for obtaining and displaying contract information" refers to means for obtaining necessary contract information from a database and displaying it so that users can verify it.

[0476] "A means of receiving user input and updating the database" refers to a means by which the system receives information entered by the user and reflects it in the database.

[0477] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing has been completed.

[0478] A "voice message" refers to voice guidance or instructions generated by generative artificial intelligence and provided to the user through speech synthesis technology.

[0479] A "server" is a computer system that processes user input information and performs tasks such as providing necessary data and updating databases.

[0480] An "AI model" is an artificial intelligence algorithm or network trained to perform data analysis and generation tasks.

[0481] This invention provides a system that allows users in unmanned stores to change contract information, enter into new contracts, and consult about plans through self-service. A specific embodiment of this system is described below.

[0482] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, and means for notifying the user of processing results.

[0483] First, upon arriving at the unmanned store, the user scans their QR code using the scanner on the provided terminal. The terminal then sends the information obtained from the QR code to a server to authenticate the user. Specifically, the terminal analyzes the data from the QR code and sends it to the server via wireless or wired communication. The server searches its database to verify the authentication information. For example, MySQL can be used as the database.

[0484] Upon successful authentication, the terminal activates the voice guidance system. The voice guidance system uses generative artificial intelligence (e.g., OpenAI's GPT-4) to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the request and determines the appropriate processing flow.

[0485] Based on the user's requirements, the server retrieves the necessary contract information and displays it on the terminal screen. For example, it might use MongoDB to retrieve contract plan information. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the necessary information according to the displayed instructions, such as their new address, desired plan, and payment method.

[0486] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[0487] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system, asking, "How can I help you?" The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which uses the generative AI model GPT-4 to determine the appropriate processing flow. The user selects their desired plan and enters their address, payment method, etc. The server updates the database and returns the processing result to the terminal. The terminal notifies the user, "Your contract is complete. Thank you for using our service."

[0488] Examples of prompt statements to input into a generative AI model are as follows:

[0489] "The user wishes to sign up for a new internet connection. Please enter the required information."

[0490] This system allows users to smoothly perform self-services in unmanned stores. This improves user convenience and enables the completion of procedures quickly and efficiently.

[0491] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0492] Step 1: User Authentication

[0493] The user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to the server to authenticate the user.

[0494] Input: The QR code held up to the scanner on the user's device.

[0495] Operation: The device scans the QR code, analyzes the data, and extracts information such as the user ID.

[0496] Data processing: Convert QR code data into text information (user ID, membership number, etc.).

[0497] Output: Text information is sent to the server.

[0498] Specific operation: The terminal sends scan data to the server via wireless or wired connection.

[0499] The server searches the database based on the received information and verifies the authentication credentials.

[0500] Input: User information sent from the device.

[0501] Data processing: Issue database queries to retrieve user information.

[0502] Output: Authentication result (success or failure).

[0503] Step 2: Activate the voice guidance system

[0504] Upon successful authentication, the terminal will activate the voice guidance system.

[0505] Operation: The device calls a speech synthesis API (e.g., Google Text-to-Speech) to generate a voice message.

[0506] Input: Authentication result status (success).

[0507] Output: A voice message "How can I help you?" is generated and played for the user.

[0508] Specific operation: The device uses a speech synthesis API to generate a voice message and plays it through the speaker.

[0509] Step 3: Enter requirements

[0510] Users enter their requirements via voice or touch panel.

[0511] Input: Requirements entered by the user via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0512] Operation: The terminal receives user input and converts it into text data.

[0513] Output: Requirements information as text data.

[0514] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice input into text data.

[0515] Step 4: Analyze requirements and determine the processing flow.

[0516] The terminal sends requirements information to the server, and the server-side AI analyzes the requirements and determines the appropriate processing flow.

[0517] Input: Text data of requirements information.

[0518] Operation: The server receives the requirements information and inputs it as prompts to a generative AI model (e.g., GPT-4).

[0519] Data processing: The AI ​​model generates a processing flow based on prompts.

[0520] Output: Processing flow of the analysis results.

[0521] Specific operation: The server calls a generative AI model and retrieves the analysis results.

[0522] Step 5: Obtain and present contract information

[0523] The server retrieves the necessary contract information based on the user's requirements and displays it on the terminal screen.

[0524] Input: Database query based on the processing flow.

[0525] Operation: The server queries the database and retrieves the necessary contract information (e.g., plan, fees, terms and conditions, etc.).

[0526] Data processing: Database queries are executed to extract contract information.

[0527] Output: Display data of contract information.

[0528] Specific operation: Data obtained from the server is sent to the terminal, and the terminal displays that information on its screen.

[0529] Step 6: Entering Information

[0530] The user follows the on-screen instructions and enters the required information (address, plan selection, payment method, etc.).

[0531] Input: Information entered by the user.

[0532] Operation: The terminal receives user input information and sends it to the server.

[0533] Output: Input information is passed to the server.

[0534] Specific operation: The device provides information as an input form and accepts input via touch panel or keyboard.

[0535] Step 7: Data Update

[0536] The server updates the database using the user's input information.

[0537] Input: User-provided information.

[0538] Operation: The server executes an update query against the database.

[0539] Data calculation: Update the corresponding record in the database.

[0540] Output: Database update results.

[0541] Specific operation: The server issues SQL queries to update database information (e.g., MongoDB or MySQL).

[0542] Step 8: Notification of processing results

[0543] Once the update is complete, the server returns the processing results to the terminal.

[0544] Input: Database update results.

[0545] Operation: The server retrieves the update results and sends them back to the terminal.

[0546] Output: Processing result (success, failure, etc.).

[0547] Specific actions: The terminal displays the result on the screen, and the voice guidance system notifies the user, "Your contract has been completed. Thank you for using our service."

[0548] By following the steps above in sequence, users can smoothly utilize self-services in unmanned stores.

[0549] (Application Example 1)

[0550] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0551] In modern society, users are required to quickly and efficiently change contract information, sign up for new contracts, and consult about plans. However, conventional systems often involve cumbersome procedures and long waiting times. In particular, in unmanned stores, it is difficult for users to perform these operations easily through self-service, and there is a need for effective means to improve the user experience. Furthermore, there is a need for a system that can be easily operated using smartphones and other mobile devices.

[0552] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0553] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, an application means installed on a smartphone, a means for collecting user requirements using voice recognition technology, and a means for determining a processing flow based on the requirements and displaying information. As a result, users can smoothly perform self-service even in unmanned stores using their smartphones, thereby reducing waiting times and improving convenience.

[0554] "User authentication means" refers to the means used to authenticate a user when they access a system.

[0555] "Generative artificial intelligence" refers to artificial intelligence that has the ability to automatically generate text and speech.

[0556] "Voice guidance means" refers to a means of guiding users using voice.

[0557] A "processing flow based on user requirements" is a means of determining the appropriate processing flow according to the requirements entered by the user.

[0558] "Contract information" refers to information about the services and plans that a user has contracted for.

[0559] "User input information" refers to all information that a user enters into the system.

[0560] "Methods for updating the database" refer to methods for updating the contents of a database based on information entered by the user.

[0561] "Means of notifying the user of the processing results" refers to means of informing the user that the processing has been completed.

[0562] "Applications installed on smartphones" refers to applications that are installed on and used on smartphones.

[0563] "Speech recognition technology" is a technology that converts speech into text.

[0564] "Means for displaying information" refers to means of displaying information on a screen or other device.

[0565] The present invention is a system for users to perform self-services using a smartphone in an unmanned store, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, means for an application installed on the smartphone, means for collecting user requirements using voice recognition technology, and means for determining a processing flow based on the requirements and displaying information.

[0566] A specific embodiment of this system will be described.

[0567] First, the user arrives at the unmanned store and scans a provided QR code using their smartphone. The user authentication system is activated, sending the information obtained from the QR code to a server to authenticate the user. If this authentication is successful, the smartphone application activates a voice guidance system using generative artificial intelligence and asks, "How can I help you?" This prompt is generated using a generative AI model.

[0568] The user inputs their requirements via voice, such as "I want to sign up for a new internet connection." Speech recognition technology, such as the Google Speech-to-Text API, converts the voice into text, and this information is sent to the server. The server analyzes the received text information and determines the processing flow based on the user's requirements.

[0569] The server retrieves contract information according to a predetermined processing flow and displays it on the smartphone screen. At this time, a voice guidance system using generative artificial intelligence guides the user through the displayed information in an easy-to-understand manner. For example, a prompt such as "Please select from these plans" may be displayed.

[0570] The user reviews the contract information and plan details displayed on the screen and enters the necessary information. The entered information is sent back to the server, which uses it to update the database. Once the update is complete, the server returns the processing result to the smartphone and notifies the user via voice guidance. For example, a prompt message such as "Your contract is complete. Thank you for using our service" may be displayed.

[0571] Here, smartphones are primarily used as hardware, and the software utilizes generative artificial intelligence models (e.g., OpenAI's GPT-4) and speech recognition technology (e.g., Google Speech-to-Text API). On the server side, cloud databases (e.g., AWS RDS or Google Cloud SQL) are used to manage contract and user information.

[0572] As a concrete example, let's look at the procedure for signing up for a new line at an unmanned store. First, the user scans a QR code, and a smartphone application launches. A voice guidance system asks, "How can I help you?", and the user replies, "I would like to sign up for a new line." The server analyzes the requirements and displays an appropriate plan, and the user enters the necessary information. After that, the contract information is updated in the database, and the result is notified to the user. This allows the user to perform the service smoothly on their own.

[0573] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0574] Step 1:

[0575] The user arrives at the unmanned store and scans a provided QR code using their smartphone. The smartphone obtains a user ID from the scanned QR code and sends it to the server as authentication information. The server uses the received user ID to look up information in its database and authenticates the user. If authentication is successful, the server sends an authentication success signal to the smartphone.

[0576] Step 2:

[0577] When the smartphone receives a successful authentication signal, a voice guidance system using generative artificial intelligence is activated. The generative AI generates the prompt "How can I help you?" and asks the user through the smartphone's voice output device. The user responds by voice, "I would like to sign up for a new line."

[0578] Step 3:

[0579] Smartphone speech recognition technology (e.g., Google Speech-to-Text API) converts the user's speech into text. This text data is sent to the server as the user's requirements. The server analyzes the received text data and determines the appropriate processing flow.

[0580] Step 4:

[0581] The server retrieves the necessary contract information from the database based on the user's requirements. The retrieved contract information is sent to the smartphone, and the contract information and plan details are displayed on the smartphone screen. Generative artificial intelligence generates prompt messages such as "Please select from these plans" and provides voice guidance.

[0582] Step 5:

[0583] The user checks their smartphone screen and enters their desired plan and necessary information. This input data is sent from the smartphone to the server. The server updates the database based on the received data and confirms that the update is complete.

[0584] Step 6:

[0585] The server records the completion of the database update and sends the result to the smartphone. The smartphone displays the processing result on the screen, and a generative artificial intelligence generates a prompt message such as "Your contract has been completed. Thank you for using our service," which is then notified to the user via voice. This allows the user to confirm the completion of the new contract.

[0586] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0587] This invention combines an emotion engine with a system that allows users to perform self-service tasks such as changing contract information, signing up for new contracts, and consulting on plans in unmanned stores. Specific embodiments of this system are described below.

[0588] This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[0589] First, the user arrives at the unmanned store terminal and scans a QR code using the terminal's scanner. The terminal sends the user information obtained from the QR code to a server for user authentication. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence.

[0590] Next, the voice guidance system, using an emotion engine, asks the user, "How can I help you?" The emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state. Based on the analyzed emotional information, the voice guidance system adjusts its response to the user's input requirements.

[0591] The user enters their requirements via voice or touch panel. For example, the requirement might be, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow.

[0592] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and the voice guidance system explains its contents to the user. During this process, the emotion engine continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[0593] The user follows the on-screen instructions and enters the necessary information (e.g., new address, desired plan, payment method, etc.). The terminal sends the information entered by the user to the server, which then updates its database based on that information.

[0594] Once the update is complete, the server sends the processing results back to the terminal. The terminal displays the processing results on the screen and notifies the user through the voice guidance system. For example, it might notify the user that "New contract completed."

[0595] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The emotion engine analyzes the user's emotional state and adjusts its response accordingly. The user replies, "I would like to sign up for a new line contract," and enters the necessary information. The server updates the database and sends the processing results back to the terminal. Finally, the terminal notifies the user that the new contract is complete, and the emotion engine displays a message to provide reassurance.

[0596] By combining these emotion engines, a system can be realized that allows users to use their services more comfortably. Flexible responses tailored to the user's emotional state become possible, leading to an expected improvement in service quality.

[0597] The following describes the processing flow.

[0598] Step 1:

[0599] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[0600] Step 2:

[0601] The device sends user information obtained from the QR code to the server.

[0602] Step 3:

[0603] The server checks the received user information against the database and performs authentication.

[0604] Step 4:

[0605] The server sends the authentication result back to the terminal.

[0606] Step 5:

[0607] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[0608] Step 6:

[0609] Through the voice guidance system, the terminal asks the user, "How can I help you?" At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[0610] Step 7:

[0611] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0612] Step 8:

[0613] The terminal sends the entered requirements to the server.

[0614] Step 9:

[0615] The server uses AI to analyze requirements and determine the appropriate processing flow.

[0616] Step 10:

[0617] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0618] Step 11:

[0619] The terminal displays acquired contract information on its screen, and a voice guidance system explains its contents to the user. Simultaneously, an emotion engine continuously monitors the user's emotions and provides messages to encourage relaxation as needed.

[0620] Step 12:

[0621] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[0622] Step 13:

[0623] The terminal sends the information entered by the user to the server.

[0624] Step 14:

[0625] The server updates the database based on the received information.

[0626] Step 15:

[0627] Once the database update is complete, the server sends the processing results back to the terminal.

[0628] Step 16:

[0629] The terminal displays the processing result on the screen and notifies the user through the voice guidance system (e.g., "Your new contract has been completed"). The emotion engine also provides messages to enhance user satisfaction based on the processing result (e.g., "Thank you for using our service. Your new contract has been successfully completed.").

[0630] Step 17:

[0631] The user presses the "Complete" button as a final confirmation to finish the process.

[0632] Step 18:

[0633] The device ends the session and displays "Thank you for using our service" on the screen. Additionally, an emotion engine monitors the user's facial expressions to confirm their satisfaction.

[0634] This series of steps, combined with an emotion engine, allows users to smoothly complete contract procedures even in unmanned stores while simultaneously receiving emotional support. This significantly improves user convenience and satisfaction.

[0635] (Example 2)

[0636] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0637] In self-service environments at unmanned stores, it is essential to provide a smooth and efficient service that minimizes user stress while allowing for changes to contract information, new contracts, and plan consultations. Furthermore, improving service quality by appropriately responding to users' emotional states is a key challenge.

[0638] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0639] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine that analyzes the user's emotions and adjusts responses according to the user's emotional state. As a result, users can use self-service in unmanned stores without feeling stressed and while receiving appropriate responses.

[0640] "User authentication means" refers to a method for a user to identify themselves at a terminal in an unmanned store and to confirm their access rights to the system.

[0641] "Voice guidance using generative artificial intelligence" refers to a method that utilizes generative artificial intelligence technology to provide guidance and answer questions through natural dialogue with the user.

[0642] "Means for determining a processing flow based on requirements" refers to the means by which the system determines the appropriate processing procedure according to the requirements entered by the user.

[0643] "Means for acquiring and displaying contract information" refers to means for retrieving necessary contract information from a database and presenting it visually to the user.

[0644] "Means for receiving user input information and updating the database" refers to means of receiving information entered by a user into a terminal and updating the database based on that information.

[0645] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing is completed.

[0646] An "emotion engine" is an engine that analyzes the user's voice tone, word choice, facial expressions, etc., to recognize the user's emotional state and adjust its response accordingly.

[0647] "Means of reading QR codes" refers to the means of reading a QR code presented by a user and obtaining its information.

[0648] "Method for sending to a server and performing authentication" refers to a method for sending the read information to a server and for the server to perform user authentication.

[0649] "Means for activating the voice guidance system" refers to the means for starting a voice guidance system using generative artificial intelligence after successful user authentication.

[0650] "Means for adjusting the voice guidance system's response based on the user's emotional state" refers to means for appropriately changing the guidance and responses of the voice guidance system according to the user's emotional state, as analyzed by the emotion engine.

[0651] Modes for carrying out the invention

[0652] This invention relates to a system for enabling users in unmanned stores to utilize self-services and smoothly change contract information, enter into new contracts, and consult about plans. This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[0653] The program for this system is processed as follows:

[0654] The hardware used includes terminals, servers, QR code readers, touch panels, microphones, speakers, and cameras installed in unmanned stores. The software includes generative artificial intelligence models, speech recognition and synthesis systems, database management systems, and sentiment analysis engines.

[0655] First, when a user arrives at the unmanned store, they scan a QR code on the terminal's scanner to authenticate themselves. The terminal sends the user information obtained from the QR code to a server, which then authenticates the user by referring to a database. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence and asks the user what they need. At this time, an emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[0656] The user enters their requirements via voice or touch panel. For example, they might enter a requirement such as, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow. The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and a voice guidance system explains the contents to the user. The emotion engine continuously monitors the user's emotional state and provides messages to promote relaxation and appropriate advice as needed.

[0657] The user follows the on-screen instructions and enters the necessary information. For example, they enter their new address, desired plan, and payment method. The terminal sends the information entered by the user to the server, which then updates its database based on that information. Once the update is complete, the server sends the processing result back to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system. For example, it might notify the user, "New contract completed," and the emotion engine might display a message to provide reassurance.

[0658] As a concrete example, a possible prompt message for a new line contract might be: "A user who wants to sign up for a new line contract has arrived at the unmanned store and scanned the QR code. What should they do next?"

[0659] This system allows users to comfortably utilize the service without experiencing stress. Furthermore, it enables flexible responses tailored to the user's emotional state, which is expected to improve the quality of the service.

[0660] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0661] Detailed explanation of the processing steps

[0662] Step 1: User Authentication Procedure

[0663] Step 1-1: The user scans the QR code.

[0664] The user arrives at the unmanned store and holds the QR code up to the terminal's scanner.

[0665] Input: QR code information

[0666] Output: QR code data

[0667] Step 1-2: The device scans the QR code.

[0668] The device reads the QR code and retrieves the data.

[0669] Input: QR code data

[0670] Output: Retrieved user information

[0671] Steps 1-3: The terminal sends the acquired user information to the server.

[0672] The terminal sends the acquired user information to the server.

[0673] Input: Retrieved user information

[0674] Output: User information sent to the server

[0675] Steps 1-4: The server performs user authentication.

[0676] The server accesses the database based on the user information it receives and performs user authentication.

[0677] Input: User information sent to the server

[0678] Data processing / calculation: User authentication via database search

[0679] Output: Authentication result (success / failure)

[0680] Step 2: Voice guidance and requirement input

[0681] Step 2-1: The terminal activates the voice guidance system.

[0682] The terminal confirms successful authentication and activates a voice guidance system using generative artificial intelligence.

[0683] Input: Authentication result (success)

[0684] Output: Activation of the voice guidance system

[0685] Step 2-2: The terminal asks the user for their requirements.

[0686] The terminal asks the user, "How can I help you?" through the voice guidance system. At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize their emotional state.

[0687] Input: Speech recognition, voice tone, word choice, facial expression

[0688] Data processing / calculation: Emotion analysis using an emotion engine

[0689] Output: User's emotional state and prompts for requirement input.

[0690] Step 3: Processing Requirements

[0691] Step 3-1: The user enters the requirements.

[0692] The user enters their requirements, such as "I want to sign up for a new line contract," via voice or touch panel.

[0693] Input: User requirements

[0694] Output: Input requirements information

[0695] Step 3-2: The terminal sends the requirements information to the server.

[0696] The terminal sends the entered requirements information to the server.

[0697] Input: Entered requirement information

[0698] Output: Requirements information sent to the server

[0699] Step 3-3: The server analyzes the requirements.

[0700] The server analyzes the information it receives and determines the appropriate processing flow.

[0701] Input: Requirements information sent to the server

[0702] Data processing / calculation: AI-based requirements analysis and processing flow determination for servers.

[0703] Output: Determined processing flow

[0704] Step 3-4: The server retrieves the contract information.

[0705] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0706] Input: Determined processing flow

[0707] Data processing / calculation: Retrieving contract information from databases

[0708] Output: Retrieved contract information

[0709] Step 4: Confirm and enter contract information

[0710] Step 4-1: The device displays the contract information.

[0711] The terminal displays acquired contract information on the screen, and a voice guidance system explains the contents to the user. An emotion engine continuously monitors the user's emotional state.

[0712] Input: Retrieved contract information

[0713] Output: Contract information displayed on the screen, voice guidance to the user.

[0714] Step 4-2: The user enters the required information.

[0715] The user follows the on-screen instructions and enters information such as their new address, desired plan, and payment method.

[0716] Input: Information required for the contract (address, plan, payment method)

[0717] Output: Input information

[0718] Step 5: Data updates and notifications

[0719] Step 5-1: The terminal sends the input information to the server.

[0720] The terminal sends the information entered by the user to the server.

[0721] Input: Information entered by the user

[0722] Output: Information sent to the server

[0723] Step 5-2: The server updates the database.

[0724] The database is updated based on the information received by the server.

[0725] Input: Information sent to the server

[0726] Data processing / calculations: Database updates

[0727] Output: Update complete result

[0728] Step 5-3: The server sends the update results to the terminal.

[0729] The server sends the result of the database update to the terminal.

[0730] Input: Update complete result

[0731] Output: Update results sent to the terminal

[0732] Step 5-4: The device notifies the user.

[0733] The terminal displays the processing result on the screen and notifies the user via the voice guidance system that "the new contract has been completed." The emotion engine then displays a message to provide reassurance.

[0734] Input: Update results sent to the device

[0735] Output: Processing results displayed on the screen, notifications to the user

[0736] By processing each step in this manner, users can utilize their own services efficiently and without stress.

[0737] (Application Example 2)

[0738] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0739] Modern self-service systems in unmanned stores and physical stores often provide only standardized responses without considering the user's emotional state, resulting in a limited user experience. This can lead to user stress or inadequate service. The present invention aims to solve this problem, enabling flexible responses based on the user's emotional state and improving the quality of service.

[0740] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0741] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, an emotion analysis means, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion feedback means. This enables the user to receive appropriate responses according to their emotional state.

[0742] "User authentication means" refers to the means used to verify that a user is legitimate when accessing a system.

[0743] "Voice guidance means using generative artificial intelligence" refers to a means of providing voice guidance using generative artificial intelligence.

[0744] An "emotion analysis method" is a means of recognizing a user's emotional state by analyzing their voice tone and facial expressions.

[0745] "Means for determining a processing flow based on user requirements" refers to a means of determining the optimal processing flow based on the requirements entered by the user.

[0746] "Means for acquiring and displaying contract information" refers to means that help acquire necessary contract information and display it to the user.

[0747] "A means of receiving user input information and updating the database" refers to a means of receiving information entered by a user and updating the database based on that information.

[0748] "Means for notifying the user of processing results" refers to the means of notifying the user of processing results obtained from the server.

[0749] An "emotional feedback method" is a means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[0750] "Means of reading QR codes" refers to any means of reading a QR code.

[0751] "A means of sending scanned information to a server and performing authentication" refers to a method of sending information obtained from a QR code to a server and performing user authentication based on that information.

[0752] "A means of receiving user requirements input by voice" refers to a means of receiving requirements entered by the user via voice.

[0753] "Means for analyzing requirements input and instructing the server on the appropriate processing flow" refers to means for analyzing the user's requirements input and instructing the server on the appropriate processing flow based on that analysis.

[0754] "Means for analyzing the user's voice tone and facial expressions" refers to means of analyzing the user's voice and facial expressions to recognize their emotional state.

[0755] "Means for displaying relaxation messages based on the user's emotional state" refers to means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[0756] The present invention is a system that allows users to change contract information, enter into new contracts, consult about plans, etc., through self-service at physical stores, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for emotion analysis, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, and means for emotion feedback.

[0757] When a user accesses a device, they first scan a QR code using the device's scanner. The scanned information is sent to a server for user authentication. This authentication process uses a standard QR code reader and an authentication system running on the server side.

[0758] Upon successful authentication, a voice guidance system using generative artificial intelligence will be activated. This system provides voice guidance as its user interface. The voice guidance uses the speech synthesis software "pyttsx3" and provides prompts to ask for the user's requirements.

[0759] Simultaneously, emotion analysis tools analyze the user's voice tone and facial expressions to recognize their emotional state. Emotion analysis utilizes facial expression recognition libraries such as "DeepFace" and voice analysis software.

[0760] When a user enters their requirements, such as "I want to sign up for a new internet connection," via voice or touch panel, that information is sent to a server. The server analyzes the requirements and determines the optimal processing flow. An AI model is used for this analysis.

[0761] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen and explains it to the user verbally. During this process, the emotion analysis system continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[0762] The user enters necessary information such as their new address, desired plan, and payment method. This information is sent to the server, and the database is updated. The server returns the processing result to the device, which displays the result on the screen and notifies the user via audio.

[0763] Furthermore, the emotional feedback system displays relaxing messages based on the user's emotional state, providing a sense of security. This allows users to complete procedures stress-free.

[0764] Specific example:

[0765] If a user says "I want to sign up for a new smartphone contract" at a physical store, the emotion analysis system can detect the user's excitement and begin the process with a message like, "It's great that you now have a new smartphone to choose from!"

[0766] Example of a prompt:

[0767] "Please suggest how to respond if a customer appears anxious."

[0768] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0769] Step 1:

[0770] The user scans a QR code with their device. The input is the data in the QR code, and the output is the user information contained in the QR code. Specifically, the device's camera captures the QR code, and the digital data is analyzed to extract the information.

[0771] Step 2:

[0772] The device sends the acquired user information to the server, which then performs authentication. The input is user information obtained from a QR code, and the output is the authentication result (success or failure). Specifically, the server verifies the user's legitimacy by comparing it with a database.

[0773] Step 3:

[0774] Once authentication on the server is successful, the terminal activates a voice guidance system using generative artificial intelligence. The input is a signal indicating successful authentication, and the output is the start of voice guidance. Specifically, the terminal uses the speech synthesis software "pyttsx3" to ask the user, "How can I help you?"

[0775] Step 4:

[0776] The voice guidance system receives user requirements via voice. The input is the user's voice, and the output is the recognized requirements. Specifically, voice recognition software converts the user's voice into text data.

[0777] Step 5:

[0778] The terminal sends the acquired voice data to the server, which analyzes the requirements and determines the processing flow. The input is text data containing the user's requirements, and the output is an appropriate processing flow. Specifically, the server uses an AI model to analyze the requirements and select the processing flow.

[0779] Step 6:

[0780] The server retrieves the necessary contract information from the database and sends it to the terminal. The input is a query containing the required contract information, and the output is the contract information itself. Specifically, the server extracts the contract information from the database and sends it back to the terminal.

[0781] Step 7:

[0782] The terminal displays contract information on its screen, and a voice guidance system explains the contents to the user. The input is contract information, and the output is voice guidance and screen display. Specifically, the terminal displays contract information on a GUI, and speech synthesis software provides explanations.

[0783] Step 8:

[0784] The emotion analysis system analyzes the user's voice tone and facial expressions to monitor their emotional state. Inputs include camera footage and microphone audio, and output is emotional state data. Specifically, emotion analysis is performed using the DeepFace library and voice analysis software.

[0785] Step 9:

[0786] The user enters the necessary information, and that data is sent from the device to the server. The input includes information provided by the user (address, plan, payment method, etc.), and the updated data is saved to the database as output. Specifically, the device accepts user input via text boxes or dropdown menus.

[0787] Step 10:

[0788] The server updates the database and sends the results back to the terminal. The input is the information to be updated, and the output is the updated result. Specifically, the server updates the database records and sends the results to the terminal.

[0789] Step 11:

[0790] The device notifies the user of the processing results, and if necessary, an emotional feedback mechanism displays a relaxation message. The input consists of processing results and emotional state data, while the output is a notification to the user and emotional feedback. Specifically, the device displays the processing results on the screen and plays a message using speech synthesis software.

[0791] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0792] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0793] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0794] [Third Embodiment]

[0795] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0796] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0797] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0798] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0799] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0800] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0801] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0802] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0803] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0804] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0805] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0806] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0807] This invention provides a system that allows users to change contract information, sign up for new contracts, and consult about plans through self-service at unmanned stores. Specific embodiments of this system are described below.

[0808] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating the database, and means for notifying the user of the processing results.

[0809] First, the user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to a server to authenticate the user. If authentication is successful, the terminal activates the voice guidance system.

[0810] The voice guidance system uses generative artificial intelligence to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to a server, where the AI ​​analyzes the request and determines the appropriate processing flow.

[0811] Next, based on the user's requirements, the necessary contract information is retrieved from the server and displayed on the terminal screen. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the required information according to the displayed instructions, such as their new address, desired plan, and payment method.

[0812] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[0813] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which determines the appropriate processing flow. The user selects their desired plan and enters the necessary information. The server then updates the database and returns the processing result to the terminal. Finally, the terminal notifies the user that the new contract has been completed.

[0814] Thus, the present invention provides a system that enables users to perform self-services smoothly even in unmanned stores. This improves user convenience and reduces waiting times.

[0815] The following describes the processing flow.

[0816] Step 1:

[0817] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[0818] Step 2:

[0819] The device sends user information obtained from the QR code to the server.

[0820] Step 3:

[0821] The server checks the received user information against the database and performs authentication.

[0822] Step 4:

[0823] The server sends the authentication result back to the terminal.

[0824] Step 5:

[0825] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[0826] Step 6:

[0827] Through the voice guidance system, the terminal will ask the user, "How can I help you?"

[0828] Step 7:

[0829] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0830] Step 8:

[0831] The terminal sends the entered requirements to the server.

[0832] Step 9:

[0833] The server uses AI to analyze requirements and determine the appropriate processing flow.

[0834] Step 10:

[0835] The server retrieves the necessary contract information from the database and sends it to the terminal.

[0836] Step 11:

[0837] The terminal displays the acquired contract information on its screen, and a voice guidance system explains the contents to the user.

[0838] Step 12:

[0839] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[0840] Step 13:

[0841] The terminal sends the information entered by the user to the server.

[0842] Step 14:

[0843] The server updates the database based on the received information.

[0844] Step 15:

[0845] Once the database update is complete, the server sends the processing results back to the terminal.

[0846] Step 16:

[0847] The terminal displays the processing result on the screen and notifies the user via the voice guidance system (e.g., "New contract completed").

[0848] Step 17:

[0849] The user presses the "Complete" button as a final confirmation to finish the process.

[0850] Step 18:

[0851] The device will end the session and display "Thank you for using our service" on the screen.

[0852] This series of steps will create a system that allows for quick and accurate contract procedures at unmanned stores.

[0853] (Example 1)

[0854] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0855] In modern society, users have a growing need to access services quickly and efficiently. However, traditional self-service stores often fail to provide adequate guidance and contract processing tailored to user needs, resulting in a poor user experience. Furthermore, understanding and completing complex procedures requires significant time and effort on the user's part, which is another major challenge.

[0856] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0857] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, a means for the voice guidance means to ask the user questions using voice messages generated by generative artificial intelligence, and a means for sending user input information to the server, where the server analyzes the input information using an AI model. This allows users to use the service easily and quickly, improving the user experience. Furthermore, complex procedures can be easily understood and completed, resulting in a reduction in time and effort.

[0858] "User authentication means" refers to the means used to verify a user's identity when they access a system.

[0859] "Generative artificial intelligence" refers to technologies that use artificial intelligence to generate text and speech in natural language processing and generative tasks.

[0860] "Voice guidance means" refers to a means of providing guidance or instructions to users using voice messages.

[0861] "Means for determining the processing flow" refers to the means by which the system selects and executes appropriate processing procedures based on user requirements.

[0862] "Means for obtaining and displaying contract information" refers to means for obtaining necessary contract information from a database and displaying it so that users can verify it.

[0863] "A means of receiving user input and updating the database" refers to a means by which the system receives information entered by the user and reflects it in the database.

[0864] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing has been completed.

[0865] A "voice message" refers to voice guidance or instructions generated by generative artificial intelligence and provided to the user through speech synthesis technology.

[0866] A "server" is a computer system that processes user input information and performs tasks such as providing necessary data and updating databases.

[0867] An "AI model" is an artificial intelligence algorithm or network trained to perform data analysis and generation tasks.

[0868] This invention provides a system that allows users in unmanned stores to change contract information, enter into new contracts, and consult about plans through self-service. A specific embodiment of this system is described below.

[0869] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, and means for notifying the user of processing results.

[0870] First, upon arriving at the unmanned store, the user scans their QR code using the scanner on the provided terminal. The terminal then sends the information obtained from the QR code to a server to authenticate the user. Specifically, the terminal analyzes the data from the QR code and sends it to the server via wireless or wired communication. The server searches its database to verify the authentication information. For example, MySQL can be used as the database.

[0871] Upon successful authentication, the terminal activates the voice guidance system. The voice guidance system uses generative artificial intelligence (e.g., OpenAI's GPT-4) to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the request and determines the appropriate processing flow.

[0872] Based on the user's requirements, the server retrieves the necessary contract information and displays it on the terminal screen. For example, it might use MongoDB to retrieve contract plan information. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the necessary information according to the displayed instructions, such as their new address, desired plan, and payment method.

[0873] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[0874] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system, asking, "How can I help you?" The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which uses the generative AI model GPT-4 to determine the appropriate processing flow. The user selects their desired plan and enters their address, payment method, etc. The server updates the database and returns the processing result to the terminal. The terminal notifies the user, "Your contract is complete. Thank you for using our service."

[0875] Examples of prompt statements to input into a generative AI model are as follows:

[0876] "The user wishes to sign up for a new internet connection. Please enter the required information."

[0877] This system allows users to smoothly perform self-services in unmanned stores. This improves user convenience and enables the completion of procedures quickly and efficiently.

[0878] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0879] Step 1: User Authentication

[0880] The user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to the server to authenticate the user.

[0881] Input: The QR code held up to the scanner on the user's device.

[0882] Operation: The device scans the QR code, analyzes the data, and extracts information such as the user ID.

[0883] Data processing: Convert QR code data into text information (user ID, membership number, etc.).

[0884] Output: Text information is sent to the server.

[0885] Specific operation: The terminal sends scan data to the server via wireless or wired connection.

[0886] The server searches the database based on the received information and verifies the authentication credentials.

[0887] Input: User information sent from the device.

[0888] Data processing: Issue database queries to retrieve user information.

[0889] Output: Authentication result (success or failure).

[0890] Step 2: Activate the voice guidance system

[0891] Upon successful authentication, the terminal will activate the voice guidance system.

[0892] Operation: The device calls a speech synthesis API (e.g., Google Text-to-Speech) to generate a voice message.

[0893] Input: Authentication result status (success).

[0894] Output: A voice message "How can I help you?" is generated and played for the user.

[0895] Specific operation: The device uses a speech synthesis API to generate a voice message and plays it through the speaker.

[0896] Step 3: Enter requirements

[0897] Users enter their requirements via voice or touch panel.

[0898] Input: Requirements entered by the user via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0899] Operation: The terminal receives user input and converts it into text data.

[0900] Output: Requirements information as text data.

[0901] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice input into text data.

[0902] Step 4: Analyze requirements and determine the processing flow.

[0903] The terminal sends requirements information to the server, and the server-side AI analyzes the requirements and determines the appropriate processing flow.

[0904] Input: Text data of requirements information.

[0905] Operation: The server receives the requirements information and inputs it as prompts to a generative AI model (e.g., GPT-4).

[0906] Data processing: The AI ​​model generates a processing flow based on prompts.

[0907] Output: Processing flow of the analysis results.

[0908] Specific operation: The server calls a generative AI model and retrieves the analysis results.

[0909] Step 5: Obtain and present contract information

[0910] The server retrieves the necessary contract information based on the user's requirements and displays it on the terminal screen.

[0911] Input: Database query based on the processing flow.

[0912] Operation: The server queries the database and retrieves the necessary contract information (e.g., plan, fees, terms and conditions, etc.).

[0913] Data processing: Database queries are executed to extract contract information.

[0914] Output: Display data of contract information.

[0915] Specific operation: Data obtained from the server is sent to the terminal, and the terminal displays that information on its screen.

[0916] Step 6: Entering Information

[0917] The user follows the on-screen instructions and enters the required information (address, plan selection, payment method, etc.).

[0918] Input: Information entered by the user.

[0919] Operation: The terminal receives user input information and sends it to the server.

[0920] Output: Input information is passed to the server.

[0921] Specific operation: The device provides information as an input form and accepts input via touch panel or keyboard.

[0922] Step 7: Data Update

[0923] The server updates the database using the user's input information.

[0924] Input: User-provided information.

[0925] Operation: The server executes an update query against the database.

[0926] Data calculation: Update the corresponding record in the database.

[0927] Output: Database update results.

[0928] Specific operation: The server issues SQL queries to update database information (e.g., MongoDB or MySQL).

[0929] Step 8: Notification of processing results

[0930] Once the update is complete, the server returns the processing results to the terminal.

[0931] Input: Database update results.

[0932] Operation: The server retrieves the update results and sends them back to the terminal.

[0933] Output: Processing result (success, failure, etc.).

[0934] Specific actions: The terminal displays the result on the screen, and the voice guidance system notifies the user, "Your contract has been completed. Thank you for using our service."

[0935] By following the steps above in sequence, users can smoothly utilize self-services in unmanned stores.

[0936] (Application Example 1)

[0937] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0938] In modern society, users are required to quickly and efficiently change contract information, sign up for new contracts, and consult about plans. However, conventional systems often involve cumbersome procedures and long waiting times. In particular, in unmanned stores, it is difficult for users to perform these operations easily through self-service, and there is a need for effective means to improve the user experience. Furthermore, there is a need for a system that can be easily operated using smartphones and other mobile devices.

[0939] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0940] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, an application means installed on a smartphone, a means for collecting user requirements using voice recognition technology, and a means for determining a processing flow based on the requirements and displaying information. As a result, users can smoothly perform self-service even in unmanned stores using their smartphones, thereby reducing waiting times and improving convenience.

[0941] "User authentication means" refers to the means used to authenticate a user when they access a system.

[0942] "Generative artificial intelligence" refers to artificial intelligence that has the ability to automatically generate text and speech.

[0943] "Voice guidance means" refers to a means of guiding users using voice.

[0944] A "processing flow based on user requirements" is a means of determining the appropriate processing flow according to the requirements entered by the user.

[0945] "Contract information" refers to information about the services and plans that a user has contracted for.

[0946] "User input information" refers to all information that a user enters into the system.

[0947] "Methods for updating the database" refer to methods for updating the contents of a database based on information entered by the user.

[0948] "Means of notifying the user of the processing results" refers to means of informing the user that the processing has been completed.

[0949] "Applications installed on smartphones" refers to applications that are installed on and used on smartphones.

[0950] "Speech recognition technology" is a technology that converts speech into text.

[0951] "Means for displaying information" refers to means of displaying information on a screen or other device.

[0952] The present invention is a system for users to perform self-services using a smartphone in an unmanned store, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, means for an application installed on the smartphone, means for collecting user requirements using voice recognition technology, and means for determining a processing flow based on the requirements and displaying information.

[0953] A specific embodiment of this system will be described.

[0954] First, the user arrives at the unmanned store and scans a provided QR code using their smartphone. The user authentication system is activated, sending the information obtained from the QR code to a server to authenticate the user. If this authentication is successful, the smartphone application activates a voice guidance system using generative artificial intelligence and asks, "How can I help you?" This prompt is generated using a generative AI model.

[0955] The user inputs their requirements via voice, such as "I want to sign up for a new internet connection." Speech recognition technology, such as the Google Speech-to-Text API, converts the voice into text, and this information is sent to the server. The server analyzes the received text information and determines the processing flow based on the user's requirements.

[0956] The server retrieves contract information according to a predetermined processing flow and displays it on the smartphone screen. At this time, a voice guidance system using generative artificial intelligence guides the user through the displayed information in an easy-to-understand manner. For example, a prompt such as "Please select from these plans" may be displayed.

[0957] The user reviews the contract information and plan details displayed on the screen and enters the necessary information. The entered information is sent back to the server, which uses it to update the database. Once the update is complete, the server returns the processing result to the smartphone and notifies the user via voice guidance. For example, a prompt message such as "Your contract is complete. Thank you for using our service" may be displayed.

[0958] Here, smartphones are primarily used as hardware, and the software utilizes generative artificial intelligence models (e.g., OpenAI's GPT-4) and speech recognition technology (e.g., Google Speech-to-Text API). On the server side, cloud databases (e.g., AWS RDS or Google Cloud SQL) are used to manage contract and user information.

[0959] As a concrete example, let's look at the procedure for signing up for a new line at an unmanned store. First, the user scans a QR code, and a smartphone application launches. A voice guidance system asks, "How can I help you?", and the user replies, "I would like to sign up for a new line." The server analyzes the requirements and displays an appropriate plan, and the user enters the necessary information. After that, the contract information is updated in the database, and the result is notified to the user. This allows the user to perform the service smoothly on their own.

[0960] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0961] Step 1:

[0962] The user arrives at the unmanned store and scans a provided QR code using their smartphone. The smartphone obtains a user ID from the scanned QR code and sends it to the server as authentication information. The server uses the received user ID to look up information in its database and authenticates the user. If authentication is successful, the server sends an authentication success signal to the smartphone.

[0963] Step 2:

[0964] When the smartphone receives a successful authentication signal, a voice guidance system using generative artificial intelligence is activated. The generative AI generates the prompt "How can I help you?" and asks the user through the smartphone's voice output device. The user responds by voice, "I would like to sign up for a new line."

[0965] Step 3:

[0966] Smartphone speech recognition technology (e.g., Google Speech-to-Text API) converts the user's speech into text. This text data is sent to the server as the user's requirements. The server analyzes the received text data and determines the appropriate processing flow.

[0967] Step 4:

[0968] The server retrieves the necessary contract information from the database based on the user's requirements. The retrieved contract information is sent to the smartphone, and the contract information and plan details are displayed on the smartphone screen. Generative artificial intelligence generates prompt messages such as "Please select from these plans" and provides voice guidance.

[0969] Step 5:

[0970] The user checks their smartphone screen and enters their desired plan and necessary information. This input data is sent from the smartphone to the server. The server updates the database based on the received data and confirms that the update is complete.

[0971] Step 6:

[0972] The server records the completion of the database update and sends the result to the smartphone. The smartphone displays the processing result on the screen, and a generative artificial intelligence generates a prompt message such as "Your contract has been completed. Thank you for using our service," which is then notified to the user via voice. This allows the user to confirm the completion of the new contract.

[0973] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0974] This invention combines an emotion engine with a system that allows users to perform self-service tasks such as changing contract information, signing up for new contracts, and consulting on plans in unmanned stores. Specific embodiments of this system are described below.

[0975] This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[0976] First, the user arrives at the unmanned store terminal and scans a QR code using the terminal's scanner. The terminal sends the user information obtained from the QR code to a server for user authentication. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence.

[0977] Next, the voice guidance system, using an emotion engine, asks the user, "How can I help you?" The emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state. Based on the analyzed emotional information, the voice guidance system adjusts its response to the user's input requirements.

[0978] The user enters their requirements via voice or touch panel. For example, the requirement might be, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow.

[0979] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and the voice guidance system explains its contents to the user. During this process, the emotion engine continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[0980] The user follows the on-screen instructions and enters the necessary information (e.g., new address, desired plan, payment method, etc.). The terminal sends the information entered by the user to the server, which then updates its database based on that information.

[0981] Once the update is complete, the server sends the processing results back to the terminal. The terminal displays the processing results on the screen and notifies the user through the voice guidance system. For example, it might notify the user that "New contract completed."

[0982] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The emotion engine analyzes the user's emotional state and adjusts its response accordingly. The user replies, "I would like to sign up for a new line contract," and enters the necessary information. The server updates the database and sends the processing results back to the terminal. Finally, the terminal notifies the user that the new contract is complete, and the emotion engine displays a message to provide reassurance.

[0983] By combining these emotion engines, a system can be realized that allows users to use their services more comfortably. Flexible responses tailored to the user's emotional state become possible, leading to an expected improvement in service quality.

[0984] The following describes the processing flow.

[0985] Step 1:

[0986] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[0987] Step 2:

[0988] The device sends user information obtained from the QR code to the server.

[0989] Step 3:

[0990] The server checks the received user information against the database and performs authentication.

[0991] Step 4:

[0992] The server sends the authentication result back to the terminal.

[0993] Step 5:

[0994] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[0995] Step 6:

[0996] Through the voice guidance system, the terminal asks the user, "How can I help you?" At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[0997] Step 7:

[0998] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[0999] Step 8:

[1000] The terminal sends the entered requirements to the server.

[1001] Step 9:

[1002] The server uses AI to analyze requirements and determine the appropriate processing flow.

[1003] Step 10:

[1004] The server retrieves the necessary contract information from the database and sends it to the terminal.

[1005] Step 11:

[1006] The terminal displays acquired contract information on its screen, and a voice guidance system explains its contents to the user. Simultaneously, an emotion engine continuously monitors the user's emotions and provides messages to encourage relaxation as needed.

[1007] Step 12:

[1008] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[1009] Step 13:

[1010] The terminal sends the information entered by the user to the server.

[1011] Step 14:

[1012] The server updates the database based on the received information.

[1013] Step 15:

[1014] Once the database update is complete, the server sends the processing results back to the terminal.

[1015] Step 16:

[1016] The terminal displays the processing result on the screen and notifies the user through the voice guidance system (e.g., "Your new contract has been completed"). The emotion engine also provides messages to enhance user satisfaction based on the processing result (e.g., "Thank you for using our service. Your new contract has been successfully completed.").

[1017] Step 17:

[1018] The user presses the "Complete" button as a final confirmation to finish the process.

[1019] Step 18:

[1020] The device ends the session and displays "Thank you for using our service" on the screen. Additionally, an emotion engine monitors the user's facial expressions to confirm their satisfaction.

[1021] This series of steps, combined with an emotion engine, allows users to smoothly complete contract procedures even in unmanned stores while simultaneously receiving emotional support. This significantly improves user convenience and satisfaction.

[1022] (Example 2)

[1023] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1024] In self-service environments at unmanned stores, it is essential to provide a smooth and efficient service that minimizes user stress while allowing for changes to contract information, new contracts, and plan consultations. Furthermore, improving service quality by appropriately responding to users' emotional states is a key challenge.

[1025] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1026] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine that analyzes the user's emotions and adjusts responses according to the user's emotional state. As a result, users can use self-service in unmanned stores without feeling stressed and while receiving appropriate responses.

[1027] "User authentication means" refers to a method for a user to identify themselves at a terminal in an unmanned store and to confirm their access rights to the system.

[1028] "Voice guidance using generative artificial intelligence" refers to a method that utilizes generative artificial intelligence technology to provide guidance and answer questions through natural dialogue with the user.

[1029] "Means for determining a processing flow based on requirements" refers to the means by which the system determines the appropriate processing procedure according to the requirements entered by the user.

[1030] "Means for acquiring and displaying contract information" refers to means for retrieving necessary contract information from a database and presenting it visually to the user.

[1031] "Means for receiving user input information and updating the database" refers to means of receiving information entered by a user into a terminal and updating the database based on that information.

[1032] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing is completed.

[1033] An "emotion engine" is an engine that analyzes the user's voice tone, word choice, facial expressions, etc., to recognize the user's emotional state and adjust its response accordingly.

[1034] "Means of reading QR codes" refers to the means of reading a QR code presented by a user and obtaining its information.

[1035] "Method for sending to a server and performing authentication" refers to a method for sending the read information to a server and for the server to perform user authentication.

[1036] "Means for activating the voice guidance system" refers to the means for starting a voice guidance system using generative artificial intelligence after successful user authentication.

[1037] "Means for adjusting the voice guidance system's response based on the user's emotional state" refers to means for appropriately changing the guidance and responses of the voice guidance system according to the user's emotional state, as analyzed by the emotion engine.

[1038] Modes for carrying out the invention

[1039] This invention relates to a system for enabling users in unmanned stores to utilize self-services and smoothly change contract information, enter into new contracts, and consult about plans. This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[1040] The program for this system is processed as follows:

[1041] The hardware used includes terminals, servers, QR code readers, touch panels, microphones, speakers, and cameras installed in unmanned stores. The software includes generative artificial intelligence models, speech recognition and synthesis systems, database management systems, and sentiment analysis engines.

[1042] First, when a user arrives at the unmanned store, they scan a QR code on the terminal's scanner to authenticate themselves. The terminal sends the user information obtained from the QR code to a server, which then authenticates the user by referring to a database. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence and asks the user what they need. At this time, an emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[1043] The user enters their requirements via voice or touch panel. For example, they might enter a requirement such as, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow. The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and a voice guidance system explains the contents to the user. The emotion engine continuously monitors the user's emotional state and provides messages to promote relaxation and appropriate advice as needed.

[1044] The user follows the on-screen instructions and enters the necessary information. For example, they enter their new address, desired plan, and payment method. The terminal sends the information entered by the user to the server, which then updates its database based on that information. Once the update is complete, the server sends the processing result back to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system. For example, it might notify the user, "New contract completed," and the emotion engine might display a message to provide reassurance.

[1045] As a concrete example, a possible prompt message for a new line contract might be: "A user who wants to sign up for a new line contract has arrived at the unmanned store and scanned the QR code. What should they do next?"

[1046] This system allows users to comfortably utilize the service without experiencing stress. Furthermore, it enables flexible responses tailored to the user's emotional state, which is expected to improve the quality of the service.

[1047] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1048] Detailed explanation of the processing steps

[1049] Step 1: User Authentication Procedure

[1050] Step 1-1: The user scans the QR code.

[1051] The user arrives at the unmanned store and holds the QR code up to the terminal's scanner.

[1052] Input: QR code information

[1053] Output: QR code data

[1054] Step 1-2: The device scans the QR code.

[1055] The device reads the QR code and retrieves the data.

[1056] Input: QR code data

[1057] Output: Retrieved user information

[1058] Steps 1-3: The terminal sends the acquired user information to the server.

[1059] The terminal sends the acquired user information to the server.

[1060] Input: Retrieved user information

[1061] Output: User information sent to the server

[1062] Steps 1-4: The server performs user authentication.

[1063] The server accesses the database based on the user information it receives and performs user authentication.

[1064] Input: User information sent to the server

[1065] Data processing / calculation: User authentication via database search

[1066] Output: Authentication result (success / failure)

[1067] Step 2: Voice guidance and requirement input

[1068] Step 2-1: The terminal activates the voice guidance system.

[1069] The terminal confirms successful authentication and activates a voice guidance system using generative artificial intelligence.

[1070] Input: Authentication result (success)

[1071] Output: Activation of the voice guidance system

[1072] Step 2-2: The terminal asks the user for their requirements.

[1073] The terminal asks the user, "How can I help you?" through the voice guidance system. At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize their emotional state.

[1074] Input: Speech recognition, voice tone, word choice, facial expression

[1075] Data processing / calculation: Emotion analysis using an emotion engine

[1076] Output: User's emotional state and prompts for requirement input.

[1077] Step 3: Processing Requirements

[1078] Step 3-1: The user enters the requirements.

[1079] The user enters their requirements, such as "I want to sign up for a new line contract," via voice or touch panel.

[1080] Input: User requirements

[1081] Output: Input requirements information

[1082] Step 3-2: The terminal sends the requirements information to the server.

[1083] The terminal sends the entered requirements information to the server.

[1084] Input: Entered requirement information

[1085] Output: Requirements information sent to the server

[1086] Step 3-3: The server analyzes the requirements.

[1087] The server analyzes the information it receives and determines the appropriate processing flow.

[1088] Input: Requirements information sent to the server

[1089] Data processing / calculation: AI-based requirements analysis and processing flow determination for servers.

[1090] Output: Determined processing flow

[1091] Step 3-4: The server retrieves the contract information.

[1092] The server retrieves the necessary contract information from the database and sends it to the terminal.

[1093] Input: Determined processing flow

[1094] Data processing / calculation: Retrieving contract information from databases

[1095] Output: Retrieved contract information

[1096] Step 4: Confirm and enter contract information

[1097] Step 4-1: The device displays the contract information.

[1098] The terminal displays acquired contract information on the screen, and a voice guidance system explains the contents to the user. An emotion engine continuously monitors the user's emotional state.

[1099] Input: Retrieved contract information

[1100] Output: Contract information displayed on the screen, voice guidance to the user.

[1101] Step 4-2: The user enters the required information.

[1102] The user follows the on-screen instructions and enters information such as their new address, desired plan, and payment method.

[1103] Input: Information required for the contract (address, plan, payment method)

[1104] Output: Input information

[1105] Step 5: Data updates and notifications

[1106] Step 5-1: The terminal sends the input information to the server.

[1107] The terminal sends the information entered by the user to the server.

[1108] Input: Information entered by the user

[1109] Output: Information sent to the server

[1110] Step 5-2: The server updates the database.

[1111] The database is updated based on the information received by the server.

[1112] Input: Information sent to the server

[1113] Data processing / calculations: Database updates

[1114] Output: Update complete result

[1115] Step 5-3: The server sends the update results to the terminal.

[1116] The server sends the result of the database update to the terminal.

[1117] Input: Update complete result

[1118] Output: Update results sent to the terminal

[1119] Step 5-4: The device notifies the user.

[1120] The terminal displays the processing result on the screen and notifies the user via the voice guidance system that "the new contract has been completed." The emotion engine then displays a message to provide reassurance.

[1121] Input: Update results sent to the device

[1122] Output: Processing results displayed on the screen, notifications to the user

[1123] By processing each step in this manner, users can utilize their own services efficiently and without stress.

[1124] (Application Example 2)

[1125] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1126] Modern self-service systems in unmanned stores and physical stores often provide only standardized responses without considering the user's emotional state, resulting in a limited user experience. This can lead to user stress or inadequate service. The present invention aims to solve this problem, enabling flexible responses based on the user's emotional state and improving the quality of service.

[1127] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1128] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, an emotion analysis means, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion feedback means. This enables the user to receive appropriate responses according to their emotional state.

[1129] "User authentication means" refers to the means used to verify that a user is legitimate when accessing a system.

[1130] "Voice guidance means using generative artificial intelligence" refers to a means of providing voice guidance using generative artificial intelligence.

[1131] An "emotion analysis method" is a means of recognizing a user's emotional state by analyzing their voice tone and facial expressions.

[1132] "Means for determining a processing flow based on user requirements" refers to a means of determining the optimal processing flow based on the requirements entered by the user.

[1133] "Means for acquiring and displaying contract information" refers to means that help acquire necessary contract information and display it to the user.

[1134] "A means of receiving user input information and updating the database" refers to a means of receiving information entered by a user and updating the database based on that information.

[1135] "Means for notifying the user of processing results" refers to the means of notifying the user of processing results obtained from the server.

[1136] An "emotional feedback method" is a means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[1137] "Means of reading QR codes" refers to any means of reading a QR code.

[1138] "A means of sending scanned information to a server and performing authentication" refers to a method of sending information obtained from a QR code to a server and performing user authentication based on that information.

[1139] "A means of receiving user requirements input by voice" refers to a means of receiving requirements entered by the user via voice.

[1140] "Means for analyzing requirements input and instructing the server on the appropriate processing flow" refers to means for analyzing the user's requirements input and instructing the server on the appropriate processing flow based on that analysis.

[1141] "Means for analyzing the user's voice tone and facial expressions" refers to means of analyzing the user's voice and facial expressions to recognize their emotional state.

[1142] "Means for displaying relaxation messages based on the user's emotional state" refers to means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[1143] The present invention is a system that allows users to change contract information, enter into new contracts, consult about plans, etc., through self-service at physical stores, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for emotion analysis, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, and means for emotion feedback.

[1144] When a user accesses a device, they first scan a QR code using the device's scanner. The scanned information is sent to a server for user authentication. This authentication process uses a standard QR code reader and an authentication system running on the server side.

[1145] Upon successful authentication, a voice guidance system using generative artificial intelligence will be activated. This system provides voice guidance as its user interface. The voice guidance uses the speech synthesis software "pyttsx3" and provides prompts to ask for the user's requirements.

[1146] Simultaneously, emotion analysis tools analyze the user's voice tone and facial expressions to recognize their emotional state. Emotion analysis utilizes facial expression recognition libraries such as "DeepFace" and voice analysis software.

[1147] When a user enters their requirements, such as "I want to sign up for a new internet connection," via voice or touch panel, that information is sent to a server. The server analyzes the requirements and determines the optimal processing flow. An AI model is used for this analysis.

[1148] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen and explains it to the user verbally. During this process, the emotion analysis system continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[1149] The user enters necessary information such as their new address, desired plan, and payment method. This information is sent to the server, and the database is updated. The server returns the processing result to the device, which displays the result on the screen and notifies the user via audio.

[1150] Furthermore, the emotional feedback system displays relaxing messages based on the user's emotional state, providing a sense of security. This allows users to complete procedures stress-free.

[1151] Specific example:

[1152] If a user says "I want to sign up for a new smartphone contract" at a physical store, the emotion analysis system can detect the user's excitement and begin the process with a message like, "It's great that you now have a new smartphone to choose from!"

[1153] Example of a prompt:

[1154] "Please suggest how to respond if a customer appears anxious."

[1155] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1156] Step 1:

[1157] The user scans a QR code with their device. The input is the data in the QR code, and the output is the user information contained in the QR code. Specifically, the device's camera captures the QR code, and the digital data is analyzed to extract the information.

[1158] Step 2:

[1159] The device sends the acquired user information to the server, which then performs authentication. The input is user information obtained from a QR code, and the output is the authentication result (success or failure). Specifically, the server verifies the user's legitimacy by comparing it with a database.

[1160] Step 3:

[1161] Once authentication on the server is successful, the terminal activates a voice guidance system using generative artificial intelligence. The input is a signal indicating successful authentication, and the output is the start of voice guidance. Specifically, the terminal uses the speech synthesis software "pyttsx3" to ask the user, "How can I help you?"

[1162] Step 4:

[1163] The voice guidance system receives user requirements via voice. The input is the user's voice, and the output is the recognized requirements. Specifically, voice recognition software converts the user's voice into text data.

[1164] Step 5:

[1165] The terminal sends the acquired voice data to the server, which analyzes the requirements and determines the processing flow. The input is text data containing the user's requirements, and the output is an appropriate processing flow. Specifically, the server uses an AI model to analyze the requirements and select the processing flow.

[1166] Step 6:

[1167] The server retrieves the necessary contract information from the database and sends it to the terminal. The input is a query containing the required contract information, and the output is the contract information itself. Specifically, the server extracts the contract information from the database and sends it back to the terminal.

[1168] Step 7:

[1169] The terminal displays contract information on its screen, and a voice guidance system explains the contents to the user. The input is contract information, and the output is voice guidance and screen display. Specifically, the terminal displays contract information on a GUI, and speech synthesis software provides explanations.

[1170] Step 8:

[1171] The emotion analysis system analyzes the user's voice tone and facial expressions to monitor their emotional state. Inputs include camera footage and microphone audio, and output is emotional state data. Specifically, emotion analysis is performed using the DeepFace library and voice analysis software.

[1172] Step 9:

[1173] The user enters the necessary information, and that data is sent from the device to the server. The input includes information provided by the user (address, plan, payment method, etc.), and the updated data is saved to the database as output. Specifically, the device accepts user input via text boxes or dropdown menus.

[1174] Step 10:

[1175] The server updates the database and sends the results back to the terminal. The input is the information to be updated, and the output is the updated result. Specifically, the server updates the database records and sends the results to the terminal.

[1176] Step 11:

[1177] The device notifies the user of the processing results, and if necessary, an emotional feedback mechanism displays a relaxation message. The input consists of processing results and emotional state data, while the output is a notification to the user and emotional feedback. Specifically, the device displays the processing results on the screen and plays a message using speech synthesis software.

[1178] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1179] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1180] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1181] [Fourth Embodiment]

[1182] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1183] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1184] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1185] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1186] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1187] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1188] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1189] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1190] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1191] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1192] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1193] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1194] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1195] This invention provides a system that allows users to change contract information, sign up for new contracts, and consult about plans through self-service at unmanned stores. Specific embodiments of this system are described below.

[1196] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating the database, and means for notifying the user of the processing results.

[1197] First, the user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to a server to authenticate the user. If authentication is successful, the terminal activates the voice guidance system.

[1198] The voice guidance system uses generative artificial intelligence to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to a server, where the AI ​​analyzes the request and determines the appropriate processing flow.

[1199] Next, based on the user's requirements, the necessary contract information is retrieved from the server and displayed on the terminal screen. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the required information according to the displayed instructions, such as their new address, desired plan, and payment method.

[1200] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[1201] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which determines the appropriate processing flow. The user selects their desired plan and enters the necessary information. The server then updates the database and returns the processing result to the terminal. Finally, the terminal notifies the user that the new contract has been completed.

[1202] Thus, the present invention provides a system that enables users to perform self-services smoothly even in unmanned stores. This improves user convenience and reduces waiting times.

[1203] The following describes the processing flow.

[1204] Step 1:

[1205] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[1206] Step 2:

[1207] The device sends user information obtained from the QR code to the server.

[1208] Step 3:

[1209] The server checks the received user information against the database and performs authentication.

[1210] Step 4:

[1211] The server sends the authentication result back to the terminal.

[1212] Step 5:

[1213] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[1214] Step 6:

[1215] Through the voice guidance system, the terminal will ask the user, "How can I help you?"

[1216] Step 7:

[1217] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[1218] Step 8:

[1219] The terminal sends the entered requirements to the server.

[1220] Step 9:

[1221] The server uses AI to analyze requirements and determine the appropriate processing flow.

[1222] Step 10:

[1223] The server retrieves the necessary contract information from the database and sends it to the terminal.

[1224] Step 11:

[1225] The terminal displays the acquired contract information on its screen, and a voice guidance system explains the contents to the user.

[1226] Step 12:

[1227] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[1228] Step 13:

[1229] The terminal sends the information entered by the user to the server.

[1230] Step 14:

[1231] The server updates the database based on the received information.

[1232] Step 15:

[1233] Once the database update is complete, the server sends the processing results back to the terminal.

[1234] Step 16:

[1235] The terminal displays the processing result on the screen and notifies the user via the voice guidance system (e.g., "New contract completed").

[1236] Step 17:

[1237] The user presses the "Complete" button as a final confirmation to finish the process.

[1238] Step 18:

[1239] The device will end the session and display "Thank you for using our service" on the screen.

[1240] This series of steps will create a system that allows for quick and accurate contract procedures at unmanned stores.

[1241] (Example 1)

[1242] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1243] In modern society, users have a growing need to access services quickly and efficiently. However, traditional self-service stores often fail to provide adequate guidance and contract processing tailored to user needs, resulting in a poor user experience. Furthermore, understanding and completing complex procedures requires significant time and effort on the user's part, which is another major challenge.

[1244] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1245] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, a means for the voice guidance means to ask the user questions using voice messages generated by generative artificial intelligence, and a means for sending user input information to the server, where the server analyzes the input information using an AI model. This allows users to use the service easily and quickly, improving the user experience. Furthermore, complex procedures can be easily understood and completed, resulting in a reduction in time and effort.

[1246] "User authentication means" refers to the means used to verify a user's identity when they access a system.

[1247] "Generative artificial intelligence" refers to technologies that use artificial intelligence to generate text and speech in natural language processing and generative tasks.

[1248] "Voice guidance means" refers to a means of providing guidance or instructions to users using voice messages.

[1249] "Means for determining the processing flow" refers to the means by which the system selects and executes appropriate processing procedures based on user requirements.

[1250] "Means for obtaining and displaying contract information" refers to means for obtaining necessary contract information from a database and displaying it so that users can verify it.

[1251] "A means of receiving user input and updating the database" refers to a means by which the system receives information entered by the user and reflects it in the database.

[1252] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing has been completed.

[1253] A "voice message" refers to voice guidance or instructions generated by generative artificial intelligence and provided to the user through speech synthesis technology.

[1254] A "server" is a computer system that processes user input information and performs tasks such as providing necessary data and updating databases.

[1255] An "AI model" is an artificial intelligence algorithm or network trained to perform data analysis and generation tasks.

[1256] This invention provides a system that allows users in unmanned stores to change contract information, enter into new contracts, and consult about plans through self-service. A specific embodiment of this system is described below.

[1257] This system includes means for user authentication, means for voice guidance using generative artificial intelligence, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, and means for notifying the user of processing results.

[1258] First, upon arriving at the unmanned store, the user scans their QR code using the scanner on the provided terminal. The terminal then sends the information obtained from the QR code to a server to authenticate the user. Specifically, the terminal analyzes the data from the QR code and sends it to the server via wireless or wired communication. The server searches its database to verify the authentication information. For example, MySQL can be used as the database.

[1259] Upon successful authentication, the terminal activates the voice guidance system. The voice guidance system uses generative artificial intelligence (e.g., OpenAI's GPT-4) to ask the user, "How can I help you?" The user enters their request via voice or touch panel. For example, they might enter, "I would like to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the request and determines the appropriate processing flow.

[1260] Based on the user's requirements, the server retrieves the necessary contract information and displays it on the terminal screen. For example, it might use MongoDB to retrieve contract plan information. During this process, a voice guidance system provides clear explanations of the displayed information. The user then enters the necessary information according to the displayed instructions, such as their new address, desired plan, and payment method.

[1261] The information entered by the user is sent back from the terminal to the server, which then uses that information to update the database. Once the update is complete, the server returns the processing result to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system.

[1262] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system, asking, "How can I help you?" The user replies, "I would like to sign up for a new line contract." The terminal sends this information to the server, which uses the generative AI model GPT-4 to determine the appropriate processing flow. The user selects their desired plan and enters their address, payment method, etc. The server updates the database and returns the processing result to the terminal. The terminal notifies the user, "Your contract is complete. Thank you for using our service."

[1263] Examples of prompt statements to input into a generative AI model are as follows:

[1264] "The user wishes to sign up for a new internet connection. Please enter the required information."

[1265] This system allows users to smoothly perform self-services in unmanned stores. This improves user convenience and enables the completion of procedures quickly and efficiently.

[1266] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1267] Step 1: User Authentication

[1268] The user arrives at the unmanned store and scans their QR code using the scanner on the provided terminal. The terminal sends the information obtained from the QR code to the server to authenticate the user.

[1269] Input: The QR code held up to the scanner on the user's device.

[1270] Operation: The device scans the QR code, analyzes the data, and extracts information such as the user ID.

[1271] Data processing: Convert QR code data into text information (user ID, membership number, etc.).

[1272] Output: Text information is sent to the server.

[1273] Specific operation: The terminal sends scan data to the server via wireless or wired connection.

[1274] The server searches the database based on the received information and verifies the authentication credentials.

[1275] Input: User information sent from the device.

[1276] Data processing: Issue database queries to retrieve user information.

[1277] Output: Authentication result (success or failure).

[1278] Step 2: Activate the voice guidance system

[1279] Upon successful authentication, the terminal will activate the voice guidance system.

[1280] Operation: The device calls a speech synthesis API (e.g., Google Text-to-Speech) to generate a voice message.

[1281] Input: Authentication result status (success).

[1282] Output: A voice message "How can I help you?" is generated and played for the user.

[1283] Specific operation: The device uses a speech synthesis API to generate a voice message and plays it through the speaker.

[1284] Step 3: Enter requirements

[1285] Users enter their requirements via voice or touch panel.

[1286] Input: Requirements entered by the user via voice or touch panel (e.g., "I want to sign up for a new line contract").

[1287] Operation: The terminal receives user input and converts it into text data.

[1288] Output: Requirements information as text data.

[1289] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice input into text data.

[1290] Step 4: Analyze requirements and determine the processing flow.

[1291] The terminal sends requirements information to the server, and the server-side AI analyzes the requirements and determines the appropriate processing flow.

[1292] Input: Text data of requirements information.

[1293] Operation: The server receives the requirements information and inputs it as prompts to a generative AI model (e.g., GPT-4).

[1294] Data processing: The AI ​​model generates a processing flow based on prompts.

[1295] Output: Processing flow of the analysis results.

[1296] Specific operation: The server calls a generative AI model and retrieves the analysis results.

[1297] Step 5: Obtain and present contract information

[1298] The server retrieves the necessary contract information based on the user's requirements and displays it on the terminal screen.

[1299] Input: Database query based on the processing flow.

[1300] Operation: The server queries the database and retrieves the necessary contract information (e.g., plan, fees, terms and conditions, etc.).

[1301] Data processing: Database queries are executed to extract contract information.

[1302] Output: Display data of contract information.

[1303] Specific operation: Data obtained from the server is sent to the terminal, and the terminal displays that information on its screen.

[1304] Step 6: Entering Information

[1305] The user follows the on-screen instructions and enters the required information (address, plan selection, payment method, etc.).

[1306] Input: Information entered by the user.

[1307] Operation: The terminal receives user input information and sends it to the server.

[1308] Output: Input information is passed to the server.

[1309] Specific operation: The device provides information as an input form and accepts input via touch panel or keyboard.

[1310] Step 7: Data Update

[1311] The server updates the database using the user's input information.

[1312] Input: User-provided information.

[1313] Operation: The server executes an update query against the database.

[1314] Data calculation: Update the corresponding record in the database.

[1315] Output: Database update results.

[1316] Specific operation: The server issues SQL queries to update database information (e.g., MongoDB or MySQL).

[1317] Step 8: Notification of processing results

[1318] Once the update is complete, the server returns the processing results to the terminal.

[1319] Input: Database update results.

[1320] Operation: The server retrieves the update results and sends them back to the terminal.

[1321] Output: Processing result (success, failure, etc.).

[1322] Specific actions: The terminal displays the result on the screen, and the voice guidance system notifies the user, "Your contract has been completed. Thank you for using our service."

[1323] By following the steps above in sequence, users can smoothly utilize self-services in unmanned stores.

[1324] (Application Example 1)

[1325] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1326] In modern society, users are required to quickly and efficiently change contract information, sign up for new contracts, and consult about plans. However, conventional systems often involve cumbersome procedures and long waiting times. In particular, in unmanned stores, it is difficult for users to perform these operations easily through self-service, and there is a need for effective means to improve the user experience. Furthermore, there is a need for a system that can be easily operated using smartphones and other mobile devices.

[1327] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1328] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, an application means installed on a smartphone, a means for collecting user requirements using voice recognition technology, and a means for determining a processing flow based on the requirements and displaying information. As a result, users can smoothly perform self-service even in unmanned stores using their smartphones, thereby reducing waiting times and improving convenience.

[1329] "User authentication means" refers to the means used to authenticate a user when they access a system.

[1330] "Generative artificial intelligence" refers to artificial intelligence that has the ability to automatically generate text and speech.

[1331] "Voice guidance means" refers to a means of guiding users using voice.

[1332] A "processing flow based on user requirements" is a means of determining the appropriate processing flow according to the requirements entered by the user.

[1333] "Contract information" refers to information about the services and plans that a user has contracted for.

[1334] "User input information" refers to all information that a user enters into the system.

[1335] "Methods for updating the database" refer to methods for updating the contents of a database based on information entered by the user.

[1336] "Means of notifying the user of the processing results" refers to means of informing the user that the processing has been completed.

[1337] "Applications installed on smartphones" refers to applications that are installed on and used on smartphones.

[1338] "Speech recognition technology" is a technology that converts speech into text.

[1339] "Means for displaying information" refers to means of displaying information on a screen or other device.

[1340] The present invention is a system for users to perform self-services using a smartphone in an unmanned store, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, means for an application installed on the smartphone, means for collecting user requirements using voice recognition technology, and means for determining a processing flow based on the requirements and displaying information.

[1341] A specific embodiment of this system will be described.

[1342] First, the user arrives at the unmanned store and scans a provided QR code using their smartphone. The user authentication system is activated, sending the information obtained from the QR code to a server to authenticate the user. If this authentication is successful, the smartphone application activates a voice guidance system using generative artificial intelligence and asks, "How can I help you?" This prompt is generated using a generative AI model.

[1343] The user inputs their requirements via voice, such as "I want to sign up for a new internet connection." Speech recognition technology, such as the Google Speech-to-Text API, converts the voice into text, and this information is sent to the server. The server analyzes the received text information and determines the processing flow based on the user's requirements.

[1344] The server retrieves contract information according to a predetermined processing flow and displays it on the smartphone screen. At this time, a voice guidance system using generative artificial intelligence guides the user through the displayed information in an easy-to-understand manner. For example, a prompt such as "Please select from these plans" may be displayed.

[1345] The user reviews the contract information and plan details displayed on the screen and enters the necessary information. The entered information is sent back to the server, which uses it to update the database. Once the update is complete, the server returns the processing result to the smartphone and notifies the user via voice guidance. For example, a prompt message such as "Your contract is complete. Thank you for using our service" may be displayed.

[1346] Here, smartphones are primarily used as hardware, and the software utilizes generative artificial intelligence models (e.g., OpenAI's GPT-4) and speech recognition technology (e.g., Google Speech-to-Text API). On the server side, cloud databases (e.g., AWS RDS or Google Cloud SQL) are used to manage contract and user information.

[1347] As a concrete example, let's look at the procedure for signing up for a new line at an unmanned store. First, the user scans a QR code, and a smartphone application launches. A voice guidance system asks, "How can I help you?", and the user replies, "I would like to sign up for a new line." The server analyzes the requirements and displays an appropriate plan, and the user enters the necessary information. After that, the contract information is updated in the database, and the result is notified to the user. This allows the user to perform the service smoothly on their own.

[1348] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1349] Step 1:

[1350] The user arrives at the unmanned store and scans a provided QR code using their smartphone. The smartphone obtains a user ID from the scanned QR code and sends it to the server as authentication information. The server uses the received user ID to look up information in its database and authenticates the user. If authentication is successful, the server sends an authentication success signal to the smartphone.

[1351] Step 2:

[1352] When the smartphone receives a successful authentication signal, a voice guidance system using generative artificial intelligence is activated. The generative AI generates the prompt "How can I help you?" and asks the user through the smartphone's voice output device. The user responds by voice, "I would like to sign up for a new line."

[1353] Step 3:

[1354] Smartphone speech recognition technology (e.g., Google Speech-to-Text API) converts the user's speech into text. This text data is sent to the server as the user's requirements. The server analyzes the received text data and determines the appropriate processing flow.

[1355] Step 4:

[1356] The server retrieves the necessary contract information from the database based on the user's requirements. The retrieved contract information is sent to the smartphone, and the contract information and plan details are displayed on the smartphone screen. Generative artificial intelligence generates prompt messages such as "Please select from these plans" and provides voice guidance.

[1357] Step 5:

[1358] The user checks their smartphone screen and enters their desired plan and necessary information. This input data is sent from the smartphone to the server. The server updates the database based on the received data and confirms that the update is complete.

[1359] Step 6:

[1360] The server records the completion of the database update and sends the result to the smartphone. The smartphone displays the processing result on the screen, and a generative artificial intelligence generates a prompt message such as "Your contract has been completed. Thank you for using our service," which is then notified to the user via voice. This allows the user to confirm the completion of the new contract.

[1361] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1362] This invention combines an emotion engine with a system that allows users to perform self-service tasks such as changing contract information, signing up for new contracts, and consulting on plans in unmanned stores. Specific embodiments of this system are described below.

[1363] This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[1364] First, the user arrives at the unmanned store terminal and scans a QR code using the terminal's scanner. The terminal sends the user information obtained from the QR code to a server for user authentication. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence.

[1365] Next, the voice guidance system, using an emotion engine, asks the user, "How can I help you?" The emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state. Based on the analyzed emotional information, the voice guidance system adjusts its response to the user's input requirements.

[1366] The user enters their requirements via voice or touch panel. For example, the requirement might be, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow.

[1367] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and the voice guidance system explains its contents to the user. During this process, the emotion engine continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[1368] The user follows the on-screen instructions and enters the necessary information (e.g., new address, desired plan, payment method, etc.). The terminal sends the information entered by the user to the server, which then updates its database based on that information.

[1369] Once the update is complete, the server sends the processing results back to the terminal. The terminal displays the processing results on the screen and notifies the user through the voice guidance system. For example, it might notify the user that "New contract completed."

[1370] As a concrete example, the procedure for signing up for a new line contract is as follows: The user arrives at an unmanned store and scans a QR code. The terminal authenticates the user and activates the voice guidance system to ask for their requirements. The emotion engine analyzes the user's emotional state and adjusts its response accordingly. The user replies, "I would like to sign up for a new line contract," and enters the necessary information. The server updates the database and sends the processing results back to the terminal. Finally, the terminal notifies the user that the new contract is complete, and the emotion engine displays a message to provide reassurance.

[1371] By combining these emotion engines, a system can be realized that allows users to use their services more comfortably. Flexible responses tailored to the user's emotional state become possible, leading to an expected improvement in service quality.

[1372] The following describes the processing flow.

[1373] Step 1:

[1374] The user arrives at the terminal in the unmanned store and scans their QR code using the QR code reader installed on the terminal.

[1375] Step 2:

[1376] The device sends user information obtained from the QR code to the server.

[1377] Step 3:

[1378] The server checks the received user information against the database and performs authentication.

[1379] Step 4:

[1380] The server sends the authentication result back to the terminal.

[1381] Step 5:

[1382] If authentication is successful, the terminal will activate a voice guidance system using generative artificial intelligence.

[1383] Step 6:

[1384] Through the voice guidance system, the terminal asks the user, "How can I help you?" At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[1385] Step 7:

[1386] The user enters their requirements via voice or touch panel (e.g., "I want to sign up for a new line contract").

[1387] Step 8:

[1388] The terminal sends the entered requirements to the server.

[1389] Step 9:

[1390] The server uses AI to analyze requirements and determine the appropriate processing flow.

[1391] Step 10:

[1392] The server retrieves the necessary contract information from the database and sends it to the terminal.

[1393] Step 11:

[1394] The terminal displays acquired contract information on its screen, and a voice guidance system explains its contents to the user. Simultaneously, an emotion engine continuously monitors the user's emotions and provides messages to encourage relaxation as needed.

[1395] Step 12:

[1396] The user follows the on-screen instructions and enters the required information (e.g., new address, desired plan, payment method, etc.).

[1397] Step 13:

[1398] The terminal sends the information entered by the user to the server.

[1399] Step 14:

[1400] The server updates the database based on the received information.

[1401] Step 15:

[1402] Once the database update is complete, the server sends the processing results back to the terminal.

[1403] Step 16:

[1404] The terminal displays the processing result on the screen and notifies the user through the voice guidance system (e.g., "Your new contract has been completed"). The emotion engine also provides messages to enhance user satisfaction based on the processing result (e.g., "Thank you for using our service. Your new contract has been successfully completed.").

[1405] Step 17:

[1406] The user presses the "Complete" button as a final confirmation to finish the process.

[1407] Step 18:

[1408] The device ends the session and displays "Thank you for using our service" on the screen. Additionally, an emotion engine monitors the user's facial expressions to confirm their satisfaction.

[1409] This series of steps, combined with an emotion engine, allows users to smoothly complete contract procedures even in unmanned stores while simultaneously receiving emotional support. This significantly improves user convenience and satisfaction.

[1410] (Example 2)

[1411] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1412] In self-service environments at unmanned stores, it is essential to provide a smooth and efficient service that minimizes user stress while allowing for changes to contract information, new contracts, and plan consultations. Furthermore, improving service quality by appropriately responding to users' emotional states is a key challenge.

[1413] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1414] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine that analyzes the user's emotions and adjusts responses according to the user's emotional state. As a result, users can use self-service in unmanned stores without feeling stressed and while receiving appropriate responses.

[1415] "User authentication means" refers to a method for a user to identify themselves at a terminal in an unmanned store and to confirm their access rights to the system.

[1416] "Voice guidance using generative artificial intelligence" refers to a method that utilizes generative artificial intelligence technology to provide guidance and answer questions through natural dialogue with the user.

[1417] "Means for determining a processing flow based on requirements" refers to the means by which the system determines the appropriate processing procedure according to the requirements entered by the user.

[1418] "Means for acquiring and displaying contract information" refers to means for retrieving necessary contract information from a database and presenting it visually to the user.

[1419] "Means for receiving user input information and updating the database" refers to means of receiving information entered by a user into a terminal and updating the database based on that information.

[1420] "Means for notifying the user of the processing results" refers to means of informing the user of the results after the processing is completed.

[1421] An "emotion engine" is an engine that analyzes the user's voice tone, word choice, facial expressions, etc., to recognize the user's emotional state and adjust its response accordingly.

[1422] "Means of reading QR codes" refers to the means of reading a QR code presented by a user and obtaining its information.

[1423] "Method for sending to a server and performing authentication" refers to a method for sending the read information to a server and for the server to perform user authentication.

[1424] "Means for activating the voice guidance system" refers to the means for starting a voice guidance system using generative artificial intelligence after successful user authentication.

[1425] "Means for adjusting the voice guidance system's response based on the user's emotional state" refers to means for appropriately changing the guidance and responses of the voice guidance system according to the user's emotional state, as analyzed by the emotion engine.

[1426] Modes for carrying out the invention

[1427] This invention relates to a system for enabling users in unmanned stores to utilize self-services and smoothly change contract information, enter into new contracts, and consult about plans. This system includes a user authentication means, a voice guidance means using generative artificial intelligence, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion engine.

[1428] The program for this system is processed as follows:

[1429] The hardware used includes terminals, servers, QR code readers, touch panels, microphones, speakers, and cameras installed in unmanned stores. The software includes generative artificial intelligence models, speech recognition and synthesis systems, database management systems, and sentiment analysis engines.

[1430] First, when a user arrives at the unmanned store, they scan a QR code on the terminal's scanner to authenticate themselves. The terminal sends the user information obtained from the QR code to a server, which then authenticates the user by referring to a database. If authentication is successful, the terminal activates a voice guidance system using generative artificial intelligence and asks the user what they need. At this time, an emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize the user's emotional state.

[1431] The user enters their requirements via voice or touch panel. For example, they might enter a requirement such as, "I want to sign up for a new internet connection." The terminal sends this information to the server, where the server's AI analyzes the requirements and determines the appropriate processing flow. The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen, and a voice guidance system explains the contents to the user. The emotion engine continuously monitors the user's emotional state and provides messages to promote relaxation and appropriate advice as needed.

[1432] The user follows the on-screen instructions and enters the necessary information. For example, they enter their new address, desired plan, and payment method. The terminal sends the information entered by the user to the server, which then updates its database based on that information. Once the update is complete, the server sends the processing result back to the terminal. The terminal displays the processing result on the screen and notifies the user through the voice guidance system. For example, it might notify the user, "New contract completed," and the emotion engine might display a message to provide reassurance.

[1433] As a concrete example, a possible prompt message for a new line contract might be: "A user who wants to sign up for a new line contract has arrived at the unmanned store and scanned the QR code. What should they do next?"

[1434] This system allows users to comfortably utilize the service without experiencing stress. Furthermore, it enables flexible responses tailored to the user's emotional state, which is expected to improve the quality of the service.

[1435] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1436] Detailed explanation of the processing steps

[1437] Step 1: User Authentication Procedure

[1438] Step 1-1: The user scans the QR code.

[1439] The user arrives at the unmanned store and holds the QR code up to the terminal's scanner.

[1440] Input: QR code information

[1441] Output: QR code data

[1442] Step 1-2: The device scans the QR code.

[1443] The device reads the QR code and retrieves the data.

[1444] Input: QR code data

[1445] Output: Retrieved user information

[1446] Steps 1-3: The terminal sends the acquired user information to the server.

[1447] The terminal sends the acquired user information to the server.

[1448] Input: Retrieved user information

[1449] Output: User information sent to the server

[1450] Steps 1-4: The server performs user authentication.

[1451] The server accesses the database based on the user information it receives and performs user authentication.

[1452] Input: User information sent to the server

[1453] Data processing / calculation: User authentication via database search

[1454] Output: Authentication result (success / failure)

[1455] Step 2: Voice guidance and requirement input

[1456] Step 2-1: The terminal activates the voice guidance system.

[1457] The terminal confirms successful authentication and activates a voice guidance system using generative artificial intelligence.

[1458] Input: Authentication result (success)

[1459] Output: Activation of the voice guidance system

[1460] Step 2-2: The terminal asks the user for their requirements.

[1461] The terminal asks the user, "How can I help you?" through the voice guidance system. At this time, the emotion engine analyzes the user's voice tone, word choice, and facial expressions to recognize their emotional state.

[1462] Input: Speech recognition, voice tone, word choice, facial expression

[1463] Data processing / calculation: Emotion analysis using an emotion engine

[1464] Output: User's emotional state and prompts for requirement input.

[1465] Step 3: Processing Requirements

[1466] Step 3-1: The user enters the requirements.

[1467] The user enters their requirements, such as "I want to sign up for a new line contract," via voice or touch panel.

[1468] Input: User requirements

[1469] Output: Input requirements information

[1470] Step 3-2: The terminal sends the requirements information to the server.

[1471] The terminal sends the entered requirements information to the server.

[1472] Input: Entered requirement information

[1473] Output: Requirements information sent to the server

[1474] Step 3-3: The server analyzes the requirements.

[1475] The server analyzes the information it receives and determines the appropriate processing flow.

[1476] Input: Requirements information sent to the server

[1477] Data processing / calculation: AI-based requirements analysis and processing flow determination for servers.

[1478] Output: Determined processing flow

[1479] Step 3-4: The server retrieves the contract information.

[1480] The server retrieves the necessary contract information from the database and sends it to the terminal.

[1481] Input: Determined processing flow

[1482] Data processing / calculation: Retrieving contract information from databases

[1483] Output: Retrieved contract information

[1484] Step 4: Confirm and enter contract information

[1485] Step 4-1: The device displays the contract information.

[1486] The terminal displays acquired contract information on the screen, and a voice guidance system explains the contents to the user. An emotion engine continuously monitors the user's emotional state.

[1487] Input: Retrieved contract information

[1488] Output: Contract information displayed on the screen, voice guidance to the user.

[1489] Step 4-2: The user enters the required information.

[1490] The user follows the on-screen instructions and enters information such as their new address, desired plan, and payment method.

[1491] Input: Information required for the contract (address, plan, payment method)

[1492] Output: Input information

[1493] Step 5: Data updates and notifications

[1494] Step 5-1: The terminal sends the input information to the server.

[1495] The terminal sends the information entered by the user to the server.

[1496] Input: Information entered by the user

[1497] Output: Information sent to the server

[1498] Step 5-2: The server updates the database.

[1499] The database is updated based on the information received by the server.

[1500] Input: Information sent to the server

[1501] Data processing / calculations: Database updates

[1502] Output: Update complete result

[1503] Step 5-3: The server sends the update results to the terminal.

[1504] The server sends the result of the database update to the terminal.

[1505] Input: Update complete result

[1506] Output: Update results sent to the terminal

[1507] Step 5-4: The device notifies the user.

[1508] The terminal displays the processing result on the screen and notifies the user via the voice guidance system that "the new contract has been completed." The emotion engine then displays a message to provide reassurance.

[1509] Input: Update results sent to the device

[1510] Output: Processing results displayed on the screen, notifications to the user

[1511] By processing each step in this manner, users can utilize their own services efficiently and without stress.

[1512] (Application Example 2)

[1513] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1514] Modern self-service systems in unmanned stores and physical stores often provide only standardized responses without considering the user's emotional state, resulting in a limited user experience. This can lead to user stress or inadequate service. The present invention aims to solve this problem, enabling flexible responses based on the user's emotional state and improving the quality of service.

[1515] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1516] In this invention, the server includes a user authentication means, a voice guidance means using generative artificial intelligence, an emotion analysis means, a means for determining a processing flow based on user requirements, a means for acquiring and displaying contract information, a means for receiving user input information and updating a database, a means for notifying the user of processing results, and an emotion feedback means. This enables the user to receive appropriate responses according to their emotional state.

[1517] "User authentication means" refers to the means used to verify that a user is legitimate when accessing a system.

[1518] "Voice guidance means using generative artificial intelligence" refers to a means of providing voice guidance using generative artificial intelligence.

[1519] An "emotion analysis method" is a means of recognizing a user's emotional state by analyzing their voice tone and facial expressions.

[1520] "Means for determining a processing flow based on user requirements" refers to a means of determining the optimal processing flow based on the requirements entered by the user.

[1521] "Means for acquiring and displaying contract information" refers to means that help acquire necessary contract information and display it to the user.

[1522] "A means of receiving user input information and updating the database" refers to a means of receiving information entered by a user and updating the database based on that information.

[1523] "Means for notifying the user of processing results" refers to the means of notifying the user of processing results obtained from the server.

[1524] An "emotional feedback method" is a means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[1525] "Means of reading QR codes" refers to any means of reading a QR code.

[1526] "A means of sending scanned information to a server and performing authentication" refers to a method of sending information obtained from a QR code to a server and performing user authentication based on that information.

[1527] "A means of receiving user requirements input by voice" refers to a means of receiving requirements entered by the user via voice.

[1528] "Means for analyzing requirements input and instructing the server on the appropriate processing flow" refers to means for analyzing the user's requirements input and instructing the server on the appropriate processing flow based on that analysis.

[1529] "Means for analyzing the user's voice tone and facial expressions" refers to means of analyzing the user's voice and facial expressions to recognize their emotional state.

[1530] "Means for displaying relaxation messages based on the user's emotional state" refers to means of displaying messages that promote relaxation and reassurance according to the user's emotional state.

[1531] The present invention is a system that allows users to change contract information, enter into new contracts, consult about plans, etc., through self-service at physical stores, and includes means for user authentication, means for voice guidance using generative artificial intelligence, means for emotion analysis, means for determining a processing flow based on user requirements, means for acquiring and displaying contract information, means for receiving user input information and updating a database, means for notifying the user of processing results, and means for emotion feedback.

[1532] When a user accesses a device, they first scan a QR code using the device's scanner. The scanned information is sent to a server for user authentication. This authentication process uses a standard QR code reader and an authentication system running on the server side.

[1533] Upon successful authentication, a voice guidance system using generative artificial intelligence will be activated. This system provides voice guidance as its user interface. The voice guidance uses the speech synthesis software "pyttsx3" and provides prompts to ask for the user's requirements.

[1534] Simultaneously, emotion analysis tools analyze the user's voice tone and facial expressions to recognize their emotional state. Emotion analysis utilizes facial expression recognition libraries such as "DeepFace" and voice analysis software.

[1535] When a user enters their requirements, such as "I want to sign up for a new internet connection," via voice or touch panel, that information is sent to a server. The server analyzes the requirements and determines the optimal processing flow. An AI model is used for this analysis.

[1536] The server retrieves the necessary contract information from the database and sends it to the terminal. The terminal displays the retrieved contract information on the screen and explains it to the user verbally. During this process, the emotion analysis system continuously monitors the user's emotional state and provides messages to encourage relaxation and appropriate advice as needed.

[1537] The user enters necessary information such as their new address, desired plan, and payment method. This information is sent to the server, and the database is updated. The server returns the processing result to the device, which displays the result on the screen and notifies the user via audio.

[1538] Furthermore, the emotional feedback system displays relaxing messages based on the user's emotional state, providing a sense of security. This allows users to complete procedures stress-free.

[1539] Specific example:

[1540] If a user says "I want to sign up for a new smartphone contract" at a physical store, the emotion analysis system can detect the user's excitement and begin the process with a message like, "It's great that you now have a new smartphone to choose from!"

[1541] Example of a prompt:

[1542] "Please suggest how to respond if a customer appears anxious."

[1543] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1544] Step 1:

[1545] The user scans a QR code with their device. The input is the data in the QR code, and the output is the user information contained in the QR code. Specifically, the device's camera captures the QR code, and the digital data is analyzed to extract the information.

[1546] Step 2:

[1547] The device sends the acquired user information to the server, which then performs authentication. The input is user information obtained from a QR code, and the output is the authentication result (success or failure). Specifically, the server verifies the user's legitimacy by comparing it with a database.

[1548] Step 3:

[1549] Once authentication on the server is successful, the terminal activates a voice guidance system using generative artificial intelligence. The input is a signal indicating successful authentication, and the output is the start of voice guidance. Specifically, the terminal uses the speech synthesis software "pyttsx3" to ask the user, "How can I help you?"

[1550] Step 4:

[1551] The voice guidance system receives user requirements via voice. The input is the user's voice, and the output is the recognized requirements. Specifically, voice recognition software converts the user's voice into text data.

[1552] Step 5:

[1553] The terminal sends the acquired voice data to the server, which analyzes the requirements and determines the processing flow. The input is text data containing the user's requirements, and the output is an appropriate processing flow. Specifically, the server uses an AI model to analyze the requirements and select the processing flow.

[1554] Step 6:

[1555] The server retrieves the necessary contract information from the database and sends it to the terminal. The input is a query containing the required contract information, and the output is the contract information itself. Specifically, the server extracts the contract information from the database and sends it back to the terminal.

[1556] Step 7:

[1557] The terminal displays contract information on its screen, and a voice guidance system explains the contents to the user. The input is contract information, and the output is voice guidance and screen display. Specifically, the terminal displays contract information on a GUI, and speech synthesis software provides explanations.

[1558] Step 8:

[1559] The emotion analysis system analyzes the user's voice tone and facial expressions to monitor their emotional state. Inputs include camera footage and microphone audio, and output is emotional state data. Specifically, emotion analysis is performed using the DeepFace library and voice analysis software.

[1560] Step 9:

[1561] The user enters the necessary information, and that data is sent from the device to the server. The input includes information provided by the user (address, plan, payment method, etc.), and the updated data is saved to the database as output. Specifically, the device accepts user input via text boxes or dropdown menus.

[1562] Step 10:

[1563] The server updates the database and sends the results back to the terminal. The input is the information to be updated, and the output is the updated result. Specifically, the server updates the database records and sends the results to the terminal.

[1564] Step 11:

[1565] The device notifies the user of the processing results, and if necessary, an emotional feedback mechanism displays a relaxation message. The input consists of processing results and emotional state data, while the output is a notification to the user and emotional feedback. Specifically, the device displays the processing results on the screen and plays a message using speech synthesis software.

[1566] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1567] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1568] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1569] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1570] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1571] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1572] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1573] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1574] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1575] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1576] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1577] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1578] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1579] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1580] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1581] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1582] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1583] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1584] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1585] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1586] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1587] The following is further disclosed regarding the embodiments described above.

[1588] (Claim 1)

[1589] User authentication method,

[1590] A voice guidance system using generative artificial intelligence,

[1591] A means of determining the processing flow based on user requirements,

[1592] Means for obtaining and displaying contract information,

[1593] A means of receiving user input information and updating the database,

[1594] A means of notifying the user of the processing results,

[1595] A system that includes this.

[1596] (Claim 2)

[1597] User authentication method,

[1598] Methods for scanning QR codes,

[1599] A means of sending the read information to the server and performing authentication,

[1600] The system according to claim 1, including the following:

[1601] (Claim 3)

[1602] A voice guidance system using generative artificial intelligence,

[1603] A means of receiving user requirements input by voice,

[1604] A means of analyzing requirements input and instructing the server on the appropriate processing flow,

[1605] The system according to claim 1, including the following:

[1606] "Example 1"

[1607] (Claim 1)

[1608] User authentication method,

[1609] A voice guidance system using generative artificial intelligence,

[1610] A means of determining the processing flow based on user requirements,

[1611] Means for obtaining and displaying contract information,

[1612] A means of receiving user input information and updating the database,

[1613] A means of notifying the user of the processing results,

[1614] A voice guidance system is a means of asking questions to the user using voice messages generated by a generative artificial intelligence system,

[1615] A means of sending user input information to a server, where the server analyzes the input information using an AI model,

[1616] A system that includes this.

[1617] (Claim 2)

[1618] Methods for scanning QR codes,

[1619] A means of sending the read information to the server and performing authentication,

[1620] A means of generating a voice message using a speech synthesis API,

[1621] The system according to claim 1, including the following:

[1622] (Claim 3)

[1623] A voice guidance system using generative artificial intelligence,

[1624] A means of receiving user requirements input by voice,

[1625] A means of analyzing requirements input and instructing the server on the appropriate processing flow,

[1626] A voice guidance system is a means of explaining contract information in detail by voice,

[1627] The system according to claim 1, including the following:

[1628] "Application Example 1"

[1629] (Claim 1)

[1630] User authentication method,

[1631] A voice guidance system using generative artificial intelligence,

[1632] A means of determining the processing flow based on user requirements,

[1633] Means for obtaining and displaying contract information,

[1634] A means of receiving user input information and updating the database,

[1635] A means of notifying the user of the processing results,

[1636] The means of application installation on a smartphone,

[1637] A means of collecting user requirements using speech recognition technology,

[1638] A means to determine the processing flow based on the requirements and to display the information,

[1639] A system that includes this.

[1640] (Claim 2)

[1641] User authentication method,

[1642] Methods for scanning QR codes,

[1643] A means of sending the read information to the server and performing authentication,

[1644] The system according to claim 1, including the following:

[1645] (Claim 3)

[1646] A voice guidance system using generative artificial intelligence,

[1647] A means of receiving user requirements input by voice,

[1648] A means of analyzing requirements input and instructing the server on the appropriate processing flow,

[1649] The system according to claim 1, including the following:

[1650] "Example 2 of combining an emotion engine"

[1651] (Claim 1)

[1652] User authentication method,

[1653] A voice guidance system using generative artificial intelligence,

[1654] A means of determining the processing flow based on user requirements,

[1655] Means for obtaining and displaying contract information,

[1656] A means of receiving user input information and updating the database,

[1657] A means of notifying the user of the processing results,

[1658] A means including an emotion engine that analyzes the user's emotions and adjusts its response according to the user's emotional state,

[1659] A system that includes this.

[1660] (Claim 2)

[1661] Methods for scanning QR codes,

[1662] A means of sending the read information to the server and performing authentication,

[1663] A means to activate the voice guidance system after successful authentication,

[1664] The system according to claim 1, including the following:

[1665] (Claim 3)

[1666] A means of recognizing the user's emotional state by analyzing their voice tone, word choice, and facial expressions,

[1667] A means of adjusting the response of the voice guidance system based on the user's emotional state,

[1668] The system according to claim 1, including the following:

[1669] "Application example 2 of combining emotional engines"

[1670] (Claim 1)

[1671] User authentication method,

[1672] A voice guidance system using generative artificial intelligence,

[1673] Emotion analysis methods,

[1674] A means of determining the processing flow based on user requirements,

[1675] Means for obtaining and displaying contract information,

[1676] A means of receiving user input information and updating the database,

[1677] A means of notifying the user of the processing results,

[1678] Emotional feedback methods,

[1679] A system that includes this.

[1680] (Claim 2)

[1681] A user authentication method that reads a QR code,

[1682] A means of sending the read information to the server and performing authentication,

[1683] The system according to claim 1, including the following:

[1684] (Claim 3)

[1685] A voice guidance system using generative artificial intelligence,

[1686] A means of receiving user requirements input by voice,

[1687] A means of analyzing requirements input and instructing the server on the appropriate processing flow,

[1688] Emotion analysis methods,

[1689] A means for analyzing the user's voice tone and facial expressions,

[1690] Emotional feedback methods are

[1691] A means of displaying a relaxation message based on the user's emotional state,

[1692] The system according to claim 1, including the following: [Explanation of symbols]

[1693] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. User authentication method, A voice guidance system using generative artificial intelligence, A means of determining the processing flow based on user requirements, Means for obtaining and displaying contract information, A means of receiving user input information and updating the database, A means of notifying the user of the processing results, A system that includes this.

2. User authentication method, Methods for scanning QR codes, A means of sending the read information to the server and performing authentication, The system according to claim 1, including the following:

3. A voice guidance system using generative artificial intelligence, A means of receiving user requirements input by voice, A means of analyzing requirements input and instructing the server on the appropriate processing flow, The system according to claim 1, including the following:

Images